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PREFACE 



This report has been prepared pursuant to the requirements of DHEW/USOE Contract Number 
300760312, Technical Assistance in Evaluatins^f'-^reer Education Projects, for^he Office of Career 
Education (OCE), This contract was awarded to NTS Research Corporation (NTS) on June 21, 1976, to 
accomphsh the following two objectives: 

1. To improve the quality of evaluations conducted by OCE-funded career education 
projects through the provision of technical assistance to fifteen of these projects. 

2. To identify, as a result of this technical assistance effort, a field tested model which 
could be used by other career c*ducation projects funded by OCE in future 
evaluation efforts. 

Contractors are encouraged to provide objective, independent evaluations; therefore, none of the 
results or positions presented in this report necessarily reflect official Office of Career Education 
policy. 

This is Volume 1 of the five volumes comprising the common Career Education Evaluation Final 
Report, The volumes include the following; 

Volume I: Summciry^ Voliimo, This volume contains highlights of. the study 
design and results. 

Volume II: The Career Education Evaluation Final Report, This, the main body 
of the report, contains detailed information about the background of the 
evaluation, the evaluation design, and the results of the evaluation. Three 
appendices are included, as well: (1) summaries of the fifteen local 
evaluations: {2) a summary of the technical assisl.mce provided by NTS; and 
(3) all instrumi^nts used in the study. 



Volume 111: The Career Education Instrumentation Suhstudy. This volume 
ntains the rationale, procedures, and results of the outcome and prix^ess 
arument validation and construct definition activities undertaken by 



cont 
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NTS under this contract . 



Volume IV: Turnkey Manual for Evaluatinfr Local Career Education Projects, 
The Turnkey Manual was prepared for distribution to participating projects 
and contains the necessary information for longitudinal continuation of 
this evaluation at the project level. 

Volume V: The Career Choice Substudy. 1'his volume contains the design and 
results of a study of the relationships of cart*er education to s^Hxmdary 
students' career choices. 

In addition to these five common report volumes, NTS prepared unique iwal evaluation reports 
for each of the fiftcn^n participating projects. 



3 



ERIC 



This mJterial in this publicaMon was prepared pursuant to a grant or contract from 
the Office of Education, U. S. Department of Health, Education, and Welfare. 
However, points of view or opinions expressed do not necessarily represent policies or 
positions of the Office of Education. 



DISCRIMINATION PROHIBITED 

Title VI of the Civil Rights Act of 1964 states: "No person in the United States shall, 
on the ground of race, color, or national origin, be excluded from participation in. be 
denied Se benefits of. or b«> subjected to discrimination under any program or activity 
receiving Federal financial assistance." Title IX of the Education Amendments of 
1972, Public Law 92-318. states: "No person in the United States shall, on the basis of 
sex. be excluded from participation in. be denied the l>enefits of. or be subjected to 
discrimination under any education program or activity recfiving Federa} financial 
assistance." Therefore, career education projects supported under Sections 402 and 
406 of the Education Amendments of 1974, like every program or activity receiving 
financial assistance from the 11. S. Department of Health, Education, and Welfare, 
must be operated in compliance with these laws. _ 
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CHAPTER I 
INTRODUCTION 

In June of 1976, the Office of Education contracted with NTS Research Corporation to provide 
technical assistance in evaluation to fifteen of their funded K-12 incremental improvement projects. 
In addition to a need to provide each project with a unique evaluation report, the Office of Career 
Education (OCR) requested that NTS also prepare a common evaluation report. This common report 
was to address the processes and outcomes of c^per education as they are manifested within these ViU 
teen projects; this study was not conceived as a national evaluation of career education. The present 
volume is a summary of the commcf^ report and includes an analysis of sei^jpf^ policy issues. 

HISTORICAL PERSPECTIVE 

The antecedents of caree^ education, as it is practiced today, can be traced to the early decades of 
this c<?ntufy. Addressing a joint meeting of the National Society for the Promotion of Industrial 
Education and National Vocational Guidance Association in 1914, John Dewey said that a 
democratic education required no "separation of vocational training from acrademic training/* His 
keynote address called for a refocusing of education to provide eaC^h learner with a * 'conception of his 
industrial obligations and opportunities." Edwin Herr's synthesis of the evolution of career 
education traces the early impetus for the career education concept to DeNvey 's seminal remarks. 

More recent landmarks include passage of the Manpower Development and Training Act of 1962, 
which explicitly recognized a need for retraining adults as technology and our economy change. With 
passage of the 1963 and 1968 vocational education legislation, emphasis shifted from the needs of the 
labor market to the employment reeds of individuals. Current federal initiatives and local interest in 
career education began in 1971, soon after Sidney P. Mariand, Jr., then Commissioner of Education, 
presented career education as a his^h priority means for educational reform. Mariand saw career 
education ae a reform effort embracing vocational educational as one significant element but also em- 
phasizing * 'occupational aspects of human development to all levels of learninp and all relevant parts 
of academic instruction" (Mariand, 1974, p, 8). 

Many factors have contributed to the current interest in carwr education as a vehicle for changing 
the direction of American education. These factors include social unrest, variations in the composition 
of the work force, dissatisfaction with schcK){ performance, and a popular desire for schools to equip 
young people for entry -level positions in the world of work. Hoyt (1975) noted that career educatioiv 
has emerged gs a weapon to attack some of these conditions calling for educational reform: 

Career education represents a response to a call for educational reform. This call has arisen 
from a variety of sources, each of \^hich has voiced dissatisfaction with American education 
as it currently exists. Suchj^ources include students, parents, the business-industry -labor 
community, out-of-school yoflth and adults, minorities, the disadvantaged, and the general 
public. While their specific concerns vary, all seem to agree that American education is in need 
of major reform at all levels. Career educatiem is properly viewed as one of several possible 
responses that could be given to this call {1975, p. 1 }. 

With increasing Siocial pressures come demailds that our schools be responsive to society's needs. 
A Gallup survey of attitudes toward the public schools revealed overwhelming agreement that schools 
should give more emphasis to the "study of trades, professions, and businesses to help students 
decide on their careers'* (Kappan, 1973), A similar survey by the American College Testing Program in 
1973 found that more than three-fourths of high school juniors expressed a need for additi6nal help 
with their career planning. In fact, career planning headed the list of preferences on the *'need for 
help" portion of the survey iMarland, 1974). It is toward these ends that career education is ad- 
dressed. « 

As a move toward consensus on the definitior^'of career education, Hoyt submit 1 xi the following 
definition in An Introduction to Career Education: A Policy Paper of the U. S. Office of Education: 



'Cwwf education' is the totality -of experiences through which one learns about and prepares 
Ip engage in wo^k as part of her o^ his way of living. 'Career. ' as defined here, is a developmen-. 
tal eoncepTlwginning in the ver>\ early years and contThtrmg well into the retirement years. 
'Educatiotf.* as defined here, ob^'iously iticludffs more than the formal education system. 
Thus, this generic definition is pufposely intended to be of a very broad and encompassing 
nature. At the'same timt?. it is intended to be less than all of life or one's reasons for living 
* (1975. pp. 2-3). • . \ ' . / 

As an evolving concept, career ^'ducatiOn has not gone without detractors. Some individuals 
claim tKat career education is little more than an affectively charged tt>rm. providing an undemanding 
focus tor educators' energies. Other criticisms are that the term hicks conceptual boundaries: It means 
many differeot things to some ptniple. and. apparently, nveans^ nothing to some. There are people who 
woyid argue that the concept comes precarioui;iy close to losing its jdentity in the larger ideological" 
entity of vocational education, or. perhaps, that there is no identity to lose, and that career education 
is what good educators ha%'e been doing for some time. ^ - 

' CAREER EDUCATION CONCEPTS AND PRiN^^ 

Hoyt (?975) stated that one.of the key objectives of education tihould be the preparation for suc- 
ceJisful working careers of all individuals.' In order to achie\ .> this ^oa\. classroom teachers have the 
responsibility of relating the material they teach in course work lo the world of work. Students must 
be given the skills which will ena'ble them to put into nractice the information they absorb in abstract 
terms. Moreo%'er. education should not l>e kept so sep irate fpom the working world that a person en- 
tering into a new career is ill-prepared to handle the human relations and unique orientation involved 
in working. " 

Preparation for lifetime careers cannot ho provided totally within the classroom environment. As 
with all learning experiences, a person is exposed to ^ variety of working roles through family mem- 
bers, community, and peers during his or her entire life. Education shoOld not be yewwi as a "begin 
and end" phase of one's experience: learning dcH's not begin at six and end at twenty-one. Also, ideally, 
a person should be free to enter and leave an educational,experience as needs and interests change. The 
.life-long conceptual framework for the individual career education may support a variety of goals. 
Depending on the given program, goal** for career education have originated from state schtwl 
superintendents' offices, hx-ul community task forces, professional educators, and a number of other 
sources. 

Goals for the elementary level 'K-B! stress awareness of the many careers available, the develop- 
ment of wholesome. attitudes of resptn-t and appreciation for the dignif.y of workers, and an awareness 
of self in relalit»n to occupational roles. A mi)re concentrated exploration of the knowledge. >kiils, 
technical requirements, working conditions, and responsibilities of various career fields comprises 
most goal statements for the middle school. These goals evolve into more specific career planning and 
preparation activities in ^jrades 9-12. . • 

Program developers and practitifirie's have formulated several levels of goals for career education: 
general goals,for all career education programs, more specific goals for individual state and schtx)! 
district efforts, and operational goals for student outcomes. In An IntroduvHon to Career Education, 
A Policy Paperofthe U. S. Office of Ediicatinju Hoyt presented nine learner outcomes (see Exhibit 1). 
which have shaped the instructi*>nal emphases in most career education pnojects.including the fifteen 
examined in this evaluation. Although NTS recognizes that this list has been superseded, these out- 
comes were operative for the p» jjects in this study. (The most recent formulation can be found jn 
Hoyt. Kenneth B.^ Primer for Career Educatiort U. S. Government Printing Office, 1977. p. 35). 

The 1975 learner outcomes were followed by this important clarification: 

It is important to note that these learner goals are intended to appi., to persons leaving the 
formal educational system for the world of work. They are not intended to be applicable 
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whinevei- the person leaves a particular sckooL For some persons, then, these goals become 
applicable when they leave the secondary school* For ot)ier$, it will be when they have left 
post*high school occupational education programs. For stiU others, the g^Mls need not be 
applied, in tofo, until they have left a college or university setting. Thus, the applicability of 
these learner outcome goals will var>' from individuj^l to individual as well as from one level of 
education to another. This is consistent with the developmental nature, and the basic assump- 
tion of individual differences inherent in the concept of career education (1975, p. 1 1). 

Exhibit 1* 

Office of Career Education Liearaer Outcomes 

Ca^-eer education seeks to produce individuals whu, when they leave school (at any age or at any level), 
are: 

1. Competent in the basic academic skills required for adapdibility in our rapidly chan^ng society. 

2. 'Equipped with good work habits. * ^ 

3. Cap€ble of choosing and who have chosen a personally meaningful set of work values that foster 
in them a desire to work. 

4. Equipped with career decisionmaking skills, job-hunting skills, and job-getting skills. 

5. Equipped with vocational personal skills at a level that will allow them to gain entry into and 
attain a degree of success in the occupational society. 

6. Equipped with career decisions based on the widest possible set of data concerning themselves 
and their educational- vocational opportunities, 

7. Aware of means available to them for continuing and recurrent education once they have left the 
formal system of schooling. 

8. Successful in being placed in a paid occupation, in further education, or in a vocation consistent 
with their current career education. ^ 

9. Successful in incorporating work values into their total personal value structure in such a way 
that they are able to choose what, for them, is a desirable lifestyle. 

*Hoyt, Kenneth' B. An Introduction to Career Education: A Policy Paper of the U. S. Office of 
Education, Washington, D. C: 1976. 



Hoyt based his definitions and concepts about career education upon the following asi^umptions. 
First, to ensure human happiness, a person must enjoy a feeling of seif-worth in work. This feeling of 
worth is not limited to the sense ont' has about his skills in performing work well, but is invcrived in all 
attitudes, values, and general abilities associated with productivity in our society. Second, a person 
develops a sense of self-worth from what is observed of the character and achievement of ethers. When 
an individual feels committed to work, and accompanying this commitment has a positive feelirf? 
toward self, the entire society of which that person is a part is directly affected. Third, the traditional 
value of the Protestant work ethic in America may not be as powerful a motivation for work as has 
previously been the case. 
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CHAPTER II 
OVERVIEW OF EVALUATION METHODOLOGY 

Foi* the purposes of this study, evaluation was defined as ''the process of clarif>'ing a set of infor- 
mation needs, and collecting, analyzing, -iid reporting information to alleviate those needs*' (Stenner, 
1973). Through a series of design conferences involving local project personnel and Office of Career 
Educatioii and NTS personnel, the evaluation tearn identified the information desired by various in- 
formation users, particularly OCE. These information needs were expressed in th6^ form of evaluation 
questions grouped around six major policy queries. The evaluation questions were subsequently 
"mapped." and the data elements required to answer each one, as well as the necessary analysis 
techniques, were specified. Finally, a comprehensive design document was prepared that set forth a 
detailed methodology for the evaluation. This chapter presents an overview of the evaluation design, 
including sectioi^^ on the conceptual framework and policy evaluation questions, methodological per- 
spectives, instrumentation, and sampling. 

CONCEPTUAL FRAMEWORK AND POLICY EVALUATION QUESTIONS 

In this section we present a conceptual framework for viewing the implementation/process of 
career education and its effects on students The framework, illustrated in Exhibit 2, represents a syn- 
thesis; of several theoretical models ^Bloom, 1976; Fullan and Pomfret, 1977; Carroll, 1963; and Cooley 
and Lohnes, 1973). Implicit in the proposed conceptual framework is an attempt to represent the 
casual factors which d'^^ermine how students perform eft the learner outcome measures. 



EXHIBIT 2 

CONCEPTUAL FRAMEWORK FOR EVALUATING FIFTEEN 
CAREER EDUCATION PROJECTS 



Learner 
Outcomes 
Pretest Scores 
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Quality of Implementation 



Determinants of 
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Career Education 
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Student variation in learner outcome posttest scores is seen as emanating from two sources: 
variation in pretest scores and variation in exposure to career education concepts and activities. 
Pretest scores serve as a proxy measure for an individual learner's fund of past experiences. In a 
review of the impor ance of entry-level performance (i.e.. pretest scores) for later learmng. BIjwm 
U976) estimates that, depending upon the length of the treatment interval, as much as one-half of 
achievement variance {i.e.. posttest performance) is attribuUble to prior experiences as manifested m 
pretest scores. Recognizing that the present evaluation study deals with an eight-month treatment m- 
terval. it is apparent that pretest scores present a potentially powerful determiner of posttest scores. 
The second source of learner outcome posttest scores, suggested by our conceptual framework, is "ex- 
posu.-e to career education" or. more specifically, the level to which a student is expose<kto career 
education concepts and activities. Variation in exposure to career education is itself a product of two 
influences: quality of implementation of career education and opportunity to learn. The level of ex- 
posure to career education exhibited by an individual student is determined partially by the level of 
teaching s^aff involvement in career education; tpacher. counselor, and principal attitudes toward 
career education; types and frequency of career education activities engaged .in by teachers, 
counselors, and principals; and other indicants of quality of implementation. The other source of 
variation in student exposure is opportunity to learn, which, at its most simplistic level, could be 
viewed as a complex of factors including motivation, attendance. ai\d attitude 'toward school. Oppor- 
tunity to learn was included to ensure conceptual completeness; cost conside»-ations. however, 
precluded operationalizing it in this study. 

Finally we attempt to e:;plain observed variation in quality of implementation by employing a set 
of determinants of implementation. Thesd^determinants. including, for example, the school district's 
experience with innovation, adoption strategies le.g,. type and frequency of inservice). and 
orgamzational capacity for change, operate to facilitate or impede the implementation of career 
education. The results should help us explain why some school districts or schools are better able to 
implement career education than others. 

Thus the conceptual framework posits a set of determinantsi which explains variation in the 
quality of implementation of career education. The quality of implementation interacts with oppor- 
tunity to learn to determine the level of exposure to career education enjoyed by a particular stodent 
U-vei of exposure to career education combined with a student's readiness, as measured by pretest 
scores, explains observed variation in posttest score performance. 

Thc^eonceptual framework suggested six policy-level evaluation questions, which formed the 
nucl^i/s for thi.s evaluation. These are: 

ID What factors determine the quality of implementation of career 
j'ducation? 

(2! Do students who have be^m expot,ed to career e iucation achieve more on 
the outcome constructs th&'n students who have not been exposed to 
career education? 

!3) In what ways are the educational experiences of career education 
' students different (both within and outside the classroom) from the 

educational experiences of noncareer education students? 

(4) Tf career education students do. in fact. learn more in the defined outcome 
domains than noncareer edqcation students, and, if the educational 
experiences of career education students are different, then in what ways 
are these experiences related to gains on the outcome constructs? 

(5) How do diffeiences in student characteristics affect the educational 
experiences found in career education programs? 
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(6) Are there differential effects of career education which interact with ) 
student characteristics? ^ ' 

"^o fully answer any one policy question, we must first answer a family of primary evaluation 
questions. Just as poiicy-making involves differentially weighing and synthesizing diverse alter- 
natives into a single perspective, answering policy evaluation questions requires a synthesis of an- 
swers to primary evaluation questions. 

METHODOLOGICAL PERSPECTIVES 

In this section we discuss six methodological issues which played majo.- roles in shaping the 
course of the evaluation. The issues are: (1) operationalizing learner outcomes, (2) implemen- 
tation/process measurement, (3) inapplicability of a true experimental design, (4) grouping items snd 
variables, (5) selecting a unit of analysis, and (6) approaches to estimating treatment effect. 

Operationalizing Learner Outcomes 

In planning for this evaluation, it quickly became apparent that available career education in- 
strumentation v/as inadequate. A thorough review revealed a paucity of well-constructed, structurally 
defined, standardized, nationally normed instruments. At least five of the nine OE learner outcomes 
(see Exhibit 1) encompassed constructs for which only limited instrumentation was available. The ab- 
sence of well-defined outcome measures required NTS to perform a major construct definition study. 
Employing several hundred items purporting to measure one or more of the outcome domains, a large- 
scale factor analysis and test validation substudy was completed. {The results of this substudy are 
described in Volume III). The construct definition study resulted in a set of outcome constructs, keyed 
to the OE learner outcomes, which possessed somewhat satisfactory psychometric properties. 

Implementation/Process Measurement 

A major focus of this evaluation was the implementation/prcxress of career education. Concep- 
tualization and meassurement of the complex career education process are difficult tasks. The dif- 
^ficulties were compounded by resource limitations which preclude any expensive, on-site obser- 
vations of classrooms and students. As a consequence, our approach to assessing the quality of im- 
plementation relied exclusively on teacher, counselor, and principal reports regarding their respective 
levels of career education knowledge and understanding, involvement in career education activities, 
and attitudes concerning career education's role and presumed benefits. Similarly, students reported 
on their frequency of "exposure" to a variety of career education activities. These activities were 
suggested by IcK'al project personnel OCE, and reviews of the literature and past evaluations of career 
education projects conducted by NTS. 

We recognize that it would have been methodologicaliv more sound to supplement our question- 
naires with classroom observation rather than rely on practitioners' reports, but the large number of 
teachers, counselors, and principals rendered this option infeasible. Another alternative to cross- 
validation was to compare each student s responses directly with his or her teacher's response so that 
reported implementation, averaged across all of a student's teachers, could be directly compared with 
student reported exposure. At the fifth grade level, this approach to cross-checking divergent perspec- 
tives on the implementation/ proi'ess proved feasible, but we felt that matching eighth and eleventh 
grade students with their respective sets of teachers across two semesters would overburden both 
participating schools and NTS data processing staff. We finally compromised as fallows: by em- 
ploying the project as the unit of aggregation it became possible to examine similarities among 
teacher, counselor, and principal reported levels of implementation and student reported exposure. 
Similarly, student reported exposure can be aggregated to the classroom level for fifth graders for 
correlation with teacher reported implementation. These approaches present some unit of analysis 
problems (see later discU3sion on this topic) but represent cost-effective options for measuring the im- 
plementation/process of career education, while, at the same time, permitting some cross-checking of 
perspectives. 
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Inapplicability of a True Experimental Design 

A major methodological issue revolved around the inapplicability of a true experimental design, 
since students were not randomly assigned to treatment and control conditions. The approach finally 
selected, which has been termed "the method of controlled correlation" (Cooley and Lohnes, 1976), 
highkghts the role of the cc-relation coefficient as the primary index of strength of relation and ex- 
planation. Numerous sets of variables are manipulated through the method of cor trolled correlation 
to answer a comprehensive list of evaluation questions. Of particular interest is the proportion of 
^ criterion variance that can be uniquely assigned to treatment group and different sets of determinants 
of implementation and implementation/process variables and the proportion of variance that is con- 
founded among various combinations of these variable sets. This approach to partitioning the 
criterion variance, turned regression/commonality analysis, is used repeatwily throughout the report. 
(A more complete discussion of the method appears in Volume II.) 

Grouping Items and Variables 

This study requires the manipulation of lar^ numbers of vejiables. In order to do this efficiently, 
the variables need to be grouped. In this study the grouping takes place at two levels: (1) components 
of the conceptual framework and (2) within each component. As it is a straightforward procedure, 
given a conceptual framework, to classify variables into the various components, this discussion will 
focus on grouping variables within components. 

The reasoning behind grouping items and variables within a component, prior to intra and inter- 
domain relationships, is a bit more subtle. The intercorrelations among knowledge and attitude items 
lead to inferences about the existence of organizing influences, which are believed to determine in- 
teritem correlations. Upon examination of these correlations, we typically make a decision to group 
highly correlated items, thus forming a scale. Weighted or unweighted combinations of item scores 
produce scale scores. It might seem logical that this same approacii should be followed in grouping 
determinant and implementation/process variables. We offer the following argument for why such an 
approach, while appropriate for grouping most outcome variables, is clearly inappropriate for 
grouping most independent or control variables. 

The fact that certain teacher characteristics are correlated with certain student characteristics, 
i.e., certain types of students are consistently taught by certain types of teachers, is generally a result 
of impUcit or explicit pohcy decisions. Such a relationship could be easily manipulated by simply 
changing the policy governing assignment of teachers to schools. The interrelationships among at- 
titude or knowledge items, on the other hand, are not so easily manipulated. It is much more difficult 
to dicUte changes in the structure of individuals' mental processes than it is to alter organizational 
patterns or coincidence of staff behaviors. Thus, we conclude that the empirically observed 
correlations among knowledge and attitude items should determine how such items are grouped: 
however, implementation/process variables should be grouped logically so that "resource packages" 
can be most easily configured and delivered. For example, a finding that the frequency with which 
counselors engage in career education activities is positively correlated with the frequency with which 
teachers engage in such activities should not lead us to group these two types of activities into a "staff 
activities" cluster. A resource package designed to increase teacher involvement in career education 
. activities would likely be quite different from one designed for counselors; thus, it is appropriate to 
distinguish between such groups of process variables when examining process-outcome relationships. 
Another example involves the division of student activities into three categories: classroom activities, 
Unkage activities, and extra-school activities.* If one set of activities is highly relate<| to an outcome. 



♦See page 21 for a further discussion of these activity variable sets>Briefly, "classroom activities" are 
those comprising part of classroom-based instruction; "linkage activities" take place outside the 
classroom but are part of formal education; and "extra-school activities" take place outside the school 
and are not part of the formal curriculum. 
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then the implicatiorf is that an appropriate change in the frequency of Uiese activities will favorably 
affect the outcome sqores. It follows that if the implementation/process variable are grouped 
logically, prior to correlating theln with outcomes, their implications for program development and 
policy will be more immediately discernible. 

Selecting a Unit of Analysis 

Sele:ting an appropriate unit of analysis is one of the most important considerations in planning 
and executing an evaluation. The purpose of this evaluation, as reflected in the policy evaluation 
questions, is to assess the effects of career education on students. For most analyses, the individual 
student, teacher, counselor, or principal will be the unit of analysis. D. R. Cox (1958} notes that the 
imit of analysis should be ''the smallest division of the experimental material such that any two units 
may receive different treatments in the actual experiment'' (p. 2). In other words, the unit of analysis 
should be the lowest level of aggregation at which units (e.g., students) are exposed to differential 
treatment. 

For the purposes of the present study, we wish to argue that the career education process exerts a 
differential effect on each individual student; there is no classroom instructional package being 
delivered which has identical effects on any two students within a classroom. Most career education 
projects claim that the career education process is implement^ throughout target schools. However, 
the level of implementation by teachers, and subsequent student reported exposure, varies widely. 
Thus, both reported implementation and student exposure to career education can be expected to vary 
from individual to individual. In short, we^argue that for most analyses the individual student, 
teacher, counselor, or principal is the appropriate unit of analysis. 

For certain questions, however, the individual student need not be the only level at which the 
results are examined. Even though students can differ in their level of exposure within a classroom, all 
students in a given classroom could receive more, or less, exposure to all or a subset of the career 
education activities than students in other classes. Teachers, as the source of much of the career 
education treatment are not, therefore, removed from the system. It is possible to aggregate fifth 
grade students within their classrooms; fifth, eighth, and eleventh grade students' results can be 
examined at the building level. 

Approaches to Estimati^ig Treatment Effect 

Three complementary approaches were used to determine the impact of career education 
processes on a range of outcome constructs. The first apprc^ch employs a comparison group to 
estimate what students would have achieved had there been no career education program. The treat- 
ment effect is equal to the difference between target and nontax get* posttest means adjusted for dif- 
ferences on the pretest. Another way of expressing the treatment effect is as the square of the partial 
correlation between group membership (target vs. nontargqt) and posttest scores. In the former case 
the treatment effect is expressed as a difference in group means, whereas in the latter the effect is 
represented by treatment condition over and above that explained by pretest variation. The F 

statistic obtained in the two cases is identical, confirming that the two approaches are statistically; as 
well as conceptually, equivalent. 

The second model tests the assumption that there is a relationship between exposure to career 
education and subs^uenfrperfonnance on the outcome constructs. If the career education program is 
having an effect, then students with high exposure to career education concepts and activities will ob- 



♦The definition and selection of target and nontarget respondents are discussed on pages 25 and 26. 



17 i ^ 



tain higher posttest scores (adjusted for pretest differences) than students with low exposure. As in 
the comparison group model there are at least two alternatives for expressing the treatment: tl) 
students could be divided into high and low exposure groups, and their respective adjusted group 
means on the outcome constr ts could be examined for differences: or (2) the exposure scores could be 
correlated with posttest scores adjusted for pretest contribution. To the extent that exposure scores 
are able to explain outcome variation beyond that explained by pretest, we infer a treatment effect. 

The third model consolidates the above two models into one analysis ^see Exhibit 3). In this model 
both group membership and exposure are correlated with criterion scores. To the extent that the 
unique contribution of group membership (segment C) to explaining criterion variation (after 
removing pretest) is large, we infer that there was some treatment effect registered on the criterion, 
but our exposure variables faUed to specify the relevant aspects of the process. If, on the other hand, 
the unique contribution of exposure variables (segment A) to explaining criterion variation is large, we 
infer that (1) the program did Httle to affect relevant process dimensions, and 12) some of the impor- 
tant process dimensions can be specified, which might be addressed by future career education 
programs interested in maximizing a particular outcome. The unique contributions of both group 
membership and exposure variables may be smaU, relative to their confounded or joint contribution 
(segment B). In this case not only has the program succeeded in generating a treatment effect, but 
the evaluation team has succeeded in specifying the particular process dimensions potentiaUy respon- 
sible for that effect. 



EXHIBITS 

BALANTINE EXPRESSING THE RELATIONSHIPS 
FOR THE JOINT EFFECTS MODEL 



Criterion 
Variance 




Exposure Treatment 
Variables Group Membership 
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INSTRUMENTATION 



The complex nature of this evaluation reHected the program's complexities and was reflected* in 
* tiini« by the wide variety of instruments used. The present summary of instrumentation draws heavily 
on Volume III, The Career Education Instrumentation Substudy. The discussion of instmmentation 
is or^nized around three respondent groups: local project staff, practitioners {teachers, rounselors. 
and principals), and students* 

Local Project Staff 

Three instruments were developed by NTS to collect information about Local Education Agen- 
cy (LEA) characteristics, program characteristics, and the extent to which career education was im- 
plemented in the project?^ A fourth instrument was developed and used in-hrnise by NTS staff to 
record appropriate census data for each project community. Table 2.1 presents s. mmary information 
describing each of these instruments. Instrument development was guided by a concern to avoid 
placing an excessive burden pn local project staff while still meeting the requirements of th:" contract 
to gather background information about each project. Project level implementation/process data are 
summarized in the following chapter; more detailed statements about each project are included in 
Volume II, Chapter 3. 



TABLE 2.1 

INSTRUMENTS FOR COLLECTING DETERMINANTS INFORMATION 



instrument 


Respondent 
Group 


Items 


Item 
Type 


Typical 
Item(s) 


LE ^ Characteristics 
F jrm 


Local 

Project 

Staff 


45 


Open 
Ended 


What has been the average num- 
ber of new faculty each year over 
the past five years^ 


Project Characteristics 
Form 


Local 

Project 

Staff 


35 


Oppn 
Ended 


Note any significant changes in, 
or additions to, program objec- 
tives over the past several years. 


Implementation/ 
Process Observation 
Form 


Local 

Project 

Staff 


35 


Open 
Ended 


Previous examples of incorpora- 
tion of large scale innovation 
within last five vears? Time for 
planning and orientation? What 
changes in role/behavior does 
the project director hope to see? 


Community Cc^nsus 
Data CheckHst 


NA 


56 


Open 
Ended 


Mobility: 
% population born in state^ 
% 1970 population living in 
same county in 1965? 
. % workers working in county of 
residence? 



Practitioners 

Although NTS developed a unique survey form for teachers, counselors, and principals, 
similarities in content and structure permit discussing these separate instruments together. The prac- 
titioner survey forms were used to collect information on the determinants of implementation and im- 
plementation/process data from each r^pondent group. Determinants of implementation included: 
staff characteristics (demographics, education experiences, and non-education career experiences); 
teaching context, counseling context, or school context; organizational capacity (morale, perceived ef- 
fectiveness); and adoption strategic (amount and kinds of inservice)* 
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TABLE 2J2 

CHARACTERISTICS OF TEACHER, COUMSELOR, AND PRINCIPAL SURVEY FORMS 
ORGANIZED BY COMPONENTS OF THE CONCEPTUAL FRAMEWORK 



VAVItABLESfT 




li«0. ef 






DETERMINANTS OP 
IMPLEMENTATiON 


TfKfiflri 


11 

9 




■1 

• Di^^ wrrtntlv twrKifS to<««rd Mtittn. Oonemw, NofMi 
vou r ud»«d or pn««r«d? fttonf . on#. ikwd. trvf««. f»wr. or 


Context Hei9 


Tfc^tfi 

^ (V^lt 


4 
4 

10 




• How pttmuiie do ^ fiM Ckir^ vour ragyltr 

• Atiout how w*tov ftwO^M* •nt MtioniO to you Hk opiiHHttinB? 

• r AtiQut M^at ow(»m«9i 01 vour tMMnn Tutc 1 tlifi^^ 




TMCti trt 
CounMUon 


2 
2 
2 


OD«n ^ DoMd Ended 


• How would vou fit* trw fftfwitl (vi^ of r«H mofSi* iOf MP^^t 
d« corps) tha piofvMtoMt tivfi in you^ ifihocM? 

ff very htgli moriffl ftmofv '^'^'^ V^^^ 

b, hi^h moral* •mono mocf of nsH 

c. mia^-KHn* higti mofalt. aomi towrnor«lt 
d low iTKKVic vmong moti of naff 

t. low wioflHi gnwifl rrn^ o< taf < 


Adopt tort Strcttgv 


TMCticri 
Counifloft 

PfifKtpAlf 


2 
2 
6 


Opvn »nd Cic»d Endtd 


• How nnany houft. tn torai. tww yov panictOit«d in ca««tr 
•ducat ton ff«H dtwikipn^tm acitviiiat finoa vow bafiftn your 
involwtwrt m c*f«ar aducttton? 

« How frcQu«niiv do faculty m«tftr>9f oonoarf««d wnh caraar 
■ducation itka p^aca in y^^ur ictHKtl^ 


PRCX:ESS DIMENSIONS 


T#»cti«i 


B 

4 ' 
4 


CiOMd Endtd ^ 


Wr1tc^ ont Qf ma loitowtfig tjan davr^bn ho«v adaquataty you aft 
(Kaearad tor tht ifn^f<n«ntM«ii of i^ratc tdifcaiton program? 

a. adaQusi«4y pf«pa«^ 

b . modarasaiy pfapMd 
e, tii^miy pfflpafad 


Duvtjom Acttviit«s 


T••c^•fl 

Principi>$ 


13 
13 
0 


of P»niCipitton 


• Advi4« fndt¥«dwa< ftudantt abou^ oir««f poisibiht<«t. 

• Giv« cftraar raialad mtrMt 3nd/Qr aptilud« tny«ntort*t to 
ffwdanti 

t. tlnKiat avary day 
b, o.TCfawtale 
c oncaor fwiot a momh 
d Mm iHan onca a rnonih 
«, nawr . 

f. daa« not a|3p4y 10 my rota 4n t fits ichooi 




1 Tractltri 

i Couniilort 


6 
1^ 
11 




• Htfppl^ and coc^tnaiaac^aar adiication fatpfor tiudanu. 

• Tmen tmih cmrwm aducat^n pfOi»ct ttatf lo pr»»M caratr 
ortcmad acttvifwi to Ftwdantt. ' 

• Conduct tffff mavttngf to pian otetf ad^t»on activiffat fo^ 
ih« ac^iool. 

■ , aimoft avtf y day 

b. oncca (»«ai( 

c. onct Of twica a month 
d ^aac than onoa ■ n>onth 
t, nawar 

r do«« not apply to my roia tnttiu lefvioi 




T«»cW*^t 

CounMtorf 
Pfinc»p«ii 


10 
12 
8 


of Piriiciptiten 


• Havantidantt 90on fiaidtr^tot>u»tnatia«and (ndufifM. 

• Uta pafantt to pratant cifatr ftiatad ptofirami to tha imdanti. 

• Spaa k to parent* and otha* €*mfTMnHy9fOupiaeKHit tht 
tmponanca o< carttr education. 

■. aimoft av^y day 

c. onct Of twioa a ^Kt^*^ 

d. (atf than pAota montft 
■ . navtr 

f. doM not app^y to my rola tn ttiii ac4^i 


Rol* tht Scft^i 


T»wefw 


40 
40 
40 


Opirtnn Itttnt 


• A caivar adueatioft program 6em mit Utanya tHa ro)a df tf^ 
couflPitOf. 

• Caraar aduoatipn promMura oaratr dofiUkKH. 

a. ftr^V^yagraa 

b. a0faa 

c. dmaffm 

d. ftronety diiaflrta 




Process dimensions were measured by three ©ets of variables: {1> knowledge and understanding of 
career educaUonM2) roIeAjefcavior related to car^r education — including classroom or counseling ac- 
tivities, linkage acUvitleis, and extra-sdhool activities; and (3) valu? internalization, as determined by 
factor analyzed responses to forty Likert-type career education opinion items. More complete descrip- 
tive information for the teacher, counselor, and principal survey fqrms is presented in Table 2.2. 

Students ' " 

Data were needed from fifth, eighth, and eleventh grade students* in 'the determinants, 
exposure, and outcome comfionents of the conceptual framework. All determinant and exposure in- 
formation, as well as a small portion of the outcome data, were collected using the Student Infor^ 
mation Form (SIF) developed by NTS. Commercially available instruments were used to gather the 
remaining outcome data. Demographic data and school experiences were considered to be the most 
relevant student level determinants of implementation for explaining exposure to career education. 
Exposure variables covered three areas: classroom activities (e.g., simulations, discussions about job 
values), linkage activities |e.g., career fairs, use of library), and extra sch(K2l activities (e.g., shadowiixg 
Experiences, work study programs). One outcome variable set, student plans and expectations, was in- 
cluded on the Student Information Form. Characteristics of the Student Information Form are 
presented in Table 2.3. 

Other instruments were required to examine learner outcomes which would (1) be suitable for pre- 
and posttesting, (2) cover the k )E learner outcomes as comprehensively as possible, and (3) be usable 
across different grade levels. Reviews of commercially available instruments uncovered a number that 
purported to meet these requirements, though no single instrument was entirely satisfactory. As time 
and resource constraints prevented development of instruments specific to the learner outcomes, NTS 
project staff selected a battery of instruments which approximated complete coverage of the con- 
structs. These instruments are described in Table 2.4. As reliability and validity data on the 
instruments were incomplete, NTS project staff determined that a complete validation study, to be 
based on pretest results, would be necessary. As validated instruments were available for the self- 
concept and basic skills outcomes, these instruments were not included in the validation study. 

Table 2.5 summarizes the administratiqn date^, respondent groups, approximate sample sizes, 
data collection methods, and data r^uction methods for all of the instruments described in this sec- 
tion. The majority of instruments were machine scorable. 

SAMPLING PLAN 

The sample consists of fifteen projects selected from the universe of K-12 incremental im- 
provement projects funded for one year by the Office of Career Education in the summer of 1976. 
Although the sample of fifteen projects was not intended to be representative of national career 
education projects, the selected projects do vary widely in their conceptualization, history, level of 
financial support, target audiences, and programmatic strategies. The sample seems to differ from a 
representative sample of the 1976 OCE projects in primarily two ways. First, more mature projects 
were given priority in the selection. Projects which had a two or three year history, funded through 
OCE or other sources, and projects with experienced staff were classified as mature. Second, only 
those projects which emphasized student learner outcomes in their proposals {as opposed to staff out- 
comes) were eligible for selection. All. fifteen projects selected for the sample agreed to^participate in 
the evaluation. Each selected LEA was assigned a' pseudonym to protect its anonymity. In this 
report, only the pseudonym is used for project identification. 



"See page 26 for an explanation of the choice of grades five, eight and eleven for this evaluation. 
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TABLE 2.3 

CHARACTtRISTiCSOF THE STUDENT INFORMATION FORM 





Ra^xmdant 


No. of 
ttams 


Item 
Type 


Typical 
IsemCd 


DETERMINANTS 
Dttwrmiiwiti of Studtnt 
EvpOiurft to CirMr 
Educition 

clwtctirlftics 
• School •xp«H«oc«s 


AlfStudant 


10 


Dosed Choice 


• Which or»a of the folloMino tM»t deicrttm 
y^r f«:edes to far in school? 

# Ho many brothers and sisters do you have? 


PROCESS 
Exposurv to CarMf 
Education Actixiti«f 

• Ctsssroom 

• Unk«ot 

• Extra SchQo\ 


Gr«j« 5 


8 


frmqxm^ of 
Participation ftemt 


• Use TV, tapes, f timstrips or redio to learn 
about jot^. 

• Take t«ts' to find itiout my lOb interests 

and skills. 

a. Almost every day 

b. Once a week 

c. Once or twice a month 

d. Less tfian once a mcmh 
a. Never 


GradaiSSi t1 


11 


GradaS 


4 


Five Oioice- • 
Frequency of 
IHirticipBtton Items 


• Attend e career f ak (a place where many 
different people come to talk about their jobs) 

• Talk about the ste» to take in making career 
decisk>ns. ^ 

a. Almost every day 

b. Once a week 

c. OfKe or twice a month ^ 

d. Lass than once a month 

e. Never 


GradaS 


8 


Grades 


7 


Five Choice- 


• Go on fteld trifM to bigness or industries to 
teern whet people do in their )obs. - 

• Take part in a sdKiol supervised ymrk 
ex$>arieoce or work-etudy program. 

a. Atnuxt every day 

b. dnceaweek 

c. Orice or twice a month 

d. lea than ofw a month 

e. Never 


Grad«$8& 11 


^ 8 


Frequency of 
Participation Items 


OUTCOME 

Plans and 
Exp«ctation$ 


Grades 


0 


aoted Choice 


• How often have you discussed your pUns 
with the following peiH>le? 

• Which one of the following best delcribes 
ptoM for the yeer aft^ you leeve high wdboot? 


GradaS 


5 




Grade 11 


10 





22 

22 



TABLE 2.4 

CHARACTERISTICS OF STUDENT LEARNER OUTCOME INSTRUMENTS 



■ " ■ \ \ 






iitffi 






















S€LF OBSERVATION 














SCAU iSOSI 
















5 ^ 


60 


yUl/no 


9 1 ihtAli fTHWt p#CK^ mfit to http <nf . 


30 


NTS 


















6 


72 








NTS 










* 1 do • k}t of.thtivi vtry MNHf 










72 




9 Otf>#r ttudtfiYf fool( to wm ^K {Mdwihif}. 


30 


NTS 










9 Compsf^ fO om fl90« 1 Morfv moY9. 






PROGRAM OF EXPLORATION 








- 






trt CAREER t:OUr:ATiON 














KNOWLrOGf TCST 






^' 










S, 11 


7b 






40 


GccrgiH Stftff 


*> 






27 T/F 


lOf t^tn^t^f , St Bf lOC^My ^ 9 M^^3l# (T/P } 




D«p«rtn>«rit of 








29 (THiU^fjitf 




















Educ#tton 










ft. wf)#r9 fVDfkvn locslcd 














c wtisrff svotkcf 1 Afs VKpvclcd to t>v loolsd in tKv fuluf9 














d. sti of tH« itKIW 




















tULIV^ilUni Tt>l ^tnita 






















SOHK foot ■Mow pCiifH# to W t'^if cf#Vt>mty 






* 












1 aatnirKi 

wW^ 1111 














Cvpoccttpn 










c. diM^rm 














d. ttrongty dtiB^rvt 








1 \ 


20 






15 


VVff t < nQt>o 










WHicti n( f^c {f)!(^Mkinf| woutd tic fh# twfi (Oiifc^ of infcHmAitOft 




Li^niffQ 










to ftfKi oul s^oul rKjuit#fTvntt sfid dMitvf ? 




Cofporstinn 










9 tiffifCd pottccfiuifi 














b. thtcfti^ of potkt 




























d. mayot <nd town counctt 








g )^ 


20 




WhtcH one ol tffc fottoMin^ scf imink wouki BEST H#!p vou 


















1 v#fn(ng 


** 












Co^poc St ton 
















> 








c. En^HfH ctud rn«fTib«f 














d. Libiarr Ciub [t>«ni<>«r 








\ \ 


20 




Wt^m you tfB fttim^ out spp^icstKsn blirak. you i^outd 


















L««« n(f>Q 


* 








■ In t^linkt iot wfifcfi you don't know jHriyvfr 














b. tn bitfnk^ tof Qu#ttton$ yvhtcf) don't A0P^y you 














C mHTO bitnk* o*i tf*t fOfPn 






i 

I * 








d in »oy tjl#rvkun#*k#d '^Of otl»c« Wft oo^y" 










2* 




Torn if M poltcwvofncr . Sh# vvntiid tikv to ssiyinctf tn h#r 


20 




\ 
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TABLE 25 

SUMMARY OF DATA COLLECTION AND REDUCTION PROCEDURES ^ 



tn«trum«nt 


Datait) 


(Wipondant 
Group 


AppfOKimata 
SampiaSita 


D«U Col(act>oit Method 


. Dau iladiictiofi Mt<hod 


ComrrHinitv C«ntus Dm%$ 
Form 


F«i3 M«rc^ 1977 


NTS Staff ' 


NA 


C«nfui Dita Rcwitw 


Hand tabuiftton and sumrnariiatton 

r 


O^tfiCttrtRics Form 


fab-March, 1977 


Proiacj Staff 




Vlatitd Quastionnaira w<th taltphpna 
foliowup 


Hand coded and tabulated 


impi^mfintatton /PfQcffU 
ObtifVitton Form 


March April, 1977 
Sapt. 1977 


Pro{ffCt Staff 


15 < 


Samntructurad on-sifa |ntarv»aw. V¥»«h 
tctaptionc fotlovvup 


Hand coded and tabulated " 


SSiMltm Infornittion Form 
I 


Spring. 1977 


Studants 


16.000 


Group adminisfraiion bv iocai ttft 
.administrators: raid to grade & ttudtntf 


Hand and rnAchme edited: matched m^xh 
{ ear ner outcorrp f lief 


' Counitlof Surwty Form 


Spring, 1977 


Coun&aioffi 


186 


CiQUd chotci quffsttQn.tjire; individual 
or group adf^inisiratto" 


Harfd and machirb edited; opinion itemi 
factor anatyxed: agoreoated to buiklmg imMi 
mtrged with student fiits 


Pfincipai Sufw«v Form 


1 

Spring. 1977 


Admtnt^fBtor& 


193 


C^o&td choica qu^Ttor^naira. individual 
or group adm{rttttration 


Hand arxj ntachir^ edited, pinion itemt 
factor anaiyzid: aggregated to bulging laiMi; 
merged vvith itud^t files 


St^f Ot>Sfrv«tton Scilet 


Fail. 1976 
Sprtn^. 1977 


Students 


lb. 000 

« 


Qroup adf^tnistfition by local tast 
^mmittrators, ra^ to grada 5 studants. 
machina rudabta formt 


Hind end machine edited, ttrtt and Mcond 
order factor anaiyied, pre-pott and Ifudtnt 

inforrnation form matcf>«d 




Fall, 1976 
Spring, 1977 


Studanti 


1 1 .000 

^> 


Proctdurat normaUy UMd in parttctpattn^ 
projects 


Hand edited; machine edited; converted to 
ttandardiitd t sccifef ; pre-po«t and stud^t 
information form nnatched 


Pro^fAfTi in Exploration of 
C*ri«r Education (PECE) 


Fail, 1975 
Sprir»g. 1977 


Studants ^ 


14.000 

( 


Group admmittratton by iocjl ttif 
admin (ft ratOfi; ra^ fo grade 5 studintf : 
fTiachtr>« raadat>U formi 


Hand edited, mechir>e adiiad, and tcored; 
. factor anaiyred, conwrted to itandardisad 
t scores, pre^post and student information 
form nuichad 


Ohio CtffiMr DiVfllOipmanT 
Tins SiritY 

Ohio 6 
Ohio 8 
Ohio 11 


Fall. 1976 
Sprms, 1977 


Studa<nts 

Grada S 
Gradt 8 


13.000 

4,000 
5,000 
4,000 


Group admmfftration by tocai tatt ^ 
«drninit«ratOfs; fMd to gracia & ftudintf ; 
rrtactiioa raadabit forms 


Hand edited, machine tdtiad. and Korod; 
factor analystd, convened to ttAndardind 
f Korof , pre-pott and student information 
form matched 


Mi«v M««icO Education lift 

NM 5001 
NMS002 
NM6006 
NM$007 


Fan. 1976 
Spr^n^. 1977 


Studanti 

Grades 5, 11 
Gf»da 11 
GradasSA 11 
Gfada 1 1 
Gradt 11 


13,000 

13.000 
4,000 
9,000 
4,000 
4,000 


Gfoup adminittration by local ftrt 
admin ittratort; raad to grade 5 studtnts; 
machtfia rgadobla forms 


Hand edited, machine editad. andfcorad; 
factor anaryaad, con«<»rfed to it«ndirdic«d 

t scores, pre-pott and ftudent tnformatioii 
form matchad 



The ext4?nt to which the fifteen career education projects selected for this evahiation are represen- 
tative of the types of activities being implemented across the nation under the banner of career 
education is difficult to determine. Several of the participa^ng school districts have been cited, eitlier » 
by their respective State Education Agencies (SEAs) or the USOE, as model districts in their past 
career education efforts. Other projects are embarking on ^ first experience with career education. 
These differences suggest diversity of program approach and maturity, but do not enlighten us on the 
representativeness of program philosophy and strategies. An admittedly- subjective opinion is that 
these projects purport to implement activities which are representative of the type5 of activities 
generally subsumed under the concept of career education. However, this study was not conceived as 
a national evaluation of career education and, consequently, little atterript was made to build represen- 
tative variation in program philosiiphies or strategies into the evaluation. Similarly, the universe of 
OCE-funded projects, from which the fifteen projects studied in this evaluation were selected, are 
themselves not a representative sample of national career education efforts. The fact that considerable 
diversity in programmatic approach exists across the fifteen projects is valuable for the study of 
process-outcome Relationships, but does not necessarily further the cause of generalizability of 
evaluation Hndings* The fifteen participating projects incorporate a wide range of program activities 
generally labeled career education activities; however, the incidence of such activities t*ationally 
remains unknown, and thus the representativeness of the fifteen projects is indeterminate. 

The fifteen LE As included in the sample vary considerably with regard to si^e of enrollment (from 
around two thousand to eighty-six thousand stiJSents}, per-pupil expenditures (from a low of just over 
seven hundred dollars to nearly eighteen h\lindted dollarslrpercentages of minority students>U|ie raoge 
is from .02 percent to seventy percent)*, and other educationally significant variables. Four of the 
projects are located in predominately rural areas, four are in* major cities, %nd the remaining seven 
projects were based in small cities and suburbs. Twelve states wPre represented by the sites; though 
only three projects were located west of the Mississippi, the rest were scattered throughout the Mid- 
west, Southeast^ and East. ' , 

Most projects attempted system wide implementation; only six targeted specific schools oi 
grades within their districts. At the beginning of the 197H'1977 school year, three of the projects 
already had five or more years of career education experience, nine projects had from four lo two 
years, two projects were starting their second year and c>nly one was just beginning the implemen- 
tation of career education. Three projects were members of their state's career education consortium. 
Although most projtH:ls embraced most of the OCE student learner outcomes in their proposals, they 
'differed in their approaches fostering student outcomes. All projects utilized staff development, 
coupled, in a few cases, to curriculum revision. Most projtKls also foi-ust»d on changes in student ac- 
tivities, including caret^r-related field trips, simulations, and career fairs. 

Following project selection, it was necessary to choOse student, samples. Students were selected 
from grades five, eijfej^t, and eleven for the **common" evaluation. To meet local evaluation needs, *a , 
variety of other grade levels were sek^*ted as well. (Irades five, eight, and eleven were selected to 
reflect ^he progression from career awareness (elementary grades) to career exploration (middle 
grades) to cari*er preparation (senior high school grades) evident in most projects. 

Originally, site administrators believed that no control students would be available in that most 
of ihe projects were implemented district-wideK^loser eij§|mination of each project revealed that con- 
trol or, more correctly, nontargeted students could be found in six of the fifteen sites. The projects 
identified these nontarget students as those whft were enrolled in schools where4here were no plans to 
inservice teachers or provide other project support. In a few cases, fifth grade students in target 
schools, but with nontargeted teachers, also were ideiTfified. Although the projects indicated that 
these nontarget -students would not differ substantially from target students across important 
' background characteristics, the data showlhat substantial differences were in fact present. Further, / 
the exposure and outcome data revealed that these non-target students received some careef/ 
education treatment and made gains on the learner outcome constructs, indicating at least some con- 
tamination. Although six sites offered nontarget students, only three sites provided substantial num- 
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bers of such students to the evaiiMition. 

\* 

Teachers, counselors, and principals were designated as target or nontarget by the projects in a 
parallel manner. If located in sch^ls not served by the project, or if not scheduled to be inserviced in 
career education by the projent^ the practitioners were defined as being nontarget. The evaluation 
team was assured that nontarget practitioners were not to be, or had not been. inservioxJ. The data, 
however, in^cate that this was not always ^he cai 3. 

The sampling design was further complicated by constraints imposed by the unique evaluation 
designs developed as part of this effort for each of the fmrticipating projects. The sampling plan had to 
ensure that adequate Mmplpg were available within each project^ as well as across projects. The r^ult 
of this constraint was a larger overall sample ^jze than would liave been nsQuired had there been no in- 
terest in within-project analyses. 

FiftJi grade classrooms were randomly selected from designated target schDols. In L£ As with 
large student populations, tatget schools were selected randomly; in small districts, usually all target 
schools were included. Eighth and eleventh grade homeroom classrooms were randomly selected from 
target junior and senior high schools. Similar selection procedures were used to draw the nontarget 
samples in the six districts in which nontarget students were available. T^le 2.6 describes the 
student samples plann«J for each of the fifteen sites and the actual sizes of the "matched" student 
samples {i.e. , those students for whom data were available from both pre- and posttesting and process 
testing). 

Within each target and nontarget school, teachers*, counselors, and administrators were sampled. 
One of three sampling schemes were employed. Either ail teachers in a school were included in the 
sample, a 33 percent random sample of teachers within a school was selected, or those teachers who 
spent most of their instructional time with the sample students were included. The latter choice was 
frequent at the high school level. Choice of sampling schemes was dictated by the project sites based 
upon administrative feasibility. All counselors in the sarnpled schools and an administrator from each 
school, usually the principal, were included in the sample. 

As Table 2.6 illustrates, the matched student samples were substantially small r than had been 
planned originally. Most of this drop-off resulted from one of three situations. Tt.j first of these, 
geographic mobility, frequently results in between fifteen and twenty percent of students tested in the 
fall .not being in attendance in the spring, Second, inclusion in the matched sample required attendan- 
ce on three separate testing dates; if a student missed any one of the three, he or she would have in- 
complex data. Delays in shipping process and posttest materials confoundai^the attendance 
problems; in a few cases necessary rescheduling was not possible. Thus, some students who had taken 
the pretest had no opportunity to take the spring tests. The third situation was confounded by the 
relatively heavy testing burden placed on students by the need to use a battery of instruments to 
cover the learner outcomes. In some projects, project staff or principals formally limited the spring 
administrations by cutting back on the sample sizes or, in a few cases, dropping the use of some in- 
struments. 



♦At the secondary level the term "teacher", as used in this report, refers to teaclfers of English, 
mathematics, science, or social studies. Teachers of physical education, music, art, and other 
peripheral subjects were not sampled in this evaluation. 




TABLE 2B 

PLANNED AND MATCHED* STUDENT SAMPLES BY PROJECT** 





■. < 

1 1_ 




Target 






Nontarget 


total 


PROJECT 


5 




8 




11 


5 




8 




11 








Planned 


Match 


Planned 


Matdi 


Piannsd 


Match 


Planned 


Match 


Planned 


Match 


Planned 




Planned 


Match 


Better Springs 






210 


405 


210 


126 










210 


145 


630 


676 


Career City 


360 


291 


360 


45S 


360 


229 


/ 

/ , 






• 






1080 


979 


Cognitive Camp 


210 


136 


390 


251 


360 


6 


210 


121 










1170 


514 


Faltstaff 


450 


328 


360 


547 


300 


120 














1110 


995 


Jobland 


450 


522 


300 


567 ' 


300 


461 














V 1150 


1550 


Middlesex 


210 


81 






















210 


81 


Mt. Exploration 


300 


(60 


420 


317 


260 


'•312 














1080 


789 


Opportunity City 


420 


119 


420 


75 


420 


82 














1260 


276 


Prepton 


210 


135 . 


240 


418 


240 


405 


210 


209 


240 


138 


240 


32 


1380 


1337 


Awaresville 










150 


7 










150 


0 


300 


7 


Studytown 


300 


256 


^0 


257 


300 


161 


300 


339 


300 


245 


300 


145 


1800 


1403 


Treksvilie 


240 


94 


210 


0 


120 


67 


240 


0 


210 


0 






' 1020 


161 


Unique Domain 


240 


126 


240 


128 


240 


88 


240 


20 


240 


71 






1290 


433 


Vocation Beach 


180 


143 


180 


97 


180 




124 














540 


3G4 


Workville 


450 


285 


^0 


263 


390 


207 














1230 


755 


TOTALS 


4020 


2676 


4020 


3784 


3930 


2395 


1200 


689 


9^ 


454 


900 


322 


15060 


10320 



''Matched samples include those for whom parallel data are available across each of the three testing periods, * 

**A blank indicates that no students were sampled at that particular grade level, usually as a result of the s^oject's programmatic emphases. Often other gr^es were 
substituted for the unique evaluations; those other gr^es are not included in this table. ' | 



CHAPTER III 
THE IMPLEMENTATION 
AND PROCESS OF C AREEREDUCATION 

This chapter summarizes our findings on the implementation/process of career education for the 
fifteen selected projects. The chapter begins by presenting an implementation/process model which 
posits a set of determinants of implementation and traces the effects of those determinants on five 
process diimensions. These process dimensions include changes in tangible resources,- intangible 
resources, knowledge and understanding, role/behavior, and value intermalization (cf. Fullan and 
Pomfret, 1977). The five process dimensions serve to define the ways in which career education is 
manifested at the school and classroom levels. Finally, consideration is given to student exposure to 
career education activities. Exposure is seen as different from student to student, and is defined as a 
product term composed of implementation quality and opportunity to learn. 

Much of our discussion on project-level determinants of implementation derives from observation 
of the fifteen projects^ review of available project documentation and periodic interaction with local 
project directors and staff over a fifteen month period. Given that there were only fifteen projects, we 
avoided use of formal measures of association and relied on frequency counts and qualitatively based 
analyses of tendency and pattern. Measurement of the five process dimensions relied on question- 
naires completed by teachers, administrators, and counselors and interviews with local project staff. 
Student exposure was operationalized through student self -reports regarding frequency of exposure 
to a broad range of career education type activities. 

AN IMPLEMENTATION/PROCESS MODEL 

A major focus of this study was the conceptualization of the implementation and process of career 
education as manifested in fifteen local projects. Conceptualizing and measuring the implemen- 
tation/process were considered important for several reasons. First, we reasoned that unless the 
process of career education, as defined by practitioners, was carefully examined, we might 
unknowingly have ended up evaluating what Charters and Jones (1973) term a **nonevent.'' The fact 
that some critics of career education were claiming no difference between **good" education and career 
education simply reinforced the importance of documenting the purported process (e.g., changes in 
resources, teacher behaviors, and administrator attitudes, etc.) of career education. 

Secondly, a review of the literature did not reveal anything like a blueprint of the career education 
process.* Although the theoretical foundation for career education was fashioned largely at the 
Federal level, local practitioners appeared to be creating and initiating alternative approaches to im- 
plementation. It quickly became apparent thay a "fidelity" perspective on the implementation of 
career education was inappropriate. We coulc^ot hope assess implementation by describing the 
discrepancies between a project's implementpffion level and some standard, since no standard existed. 
Rather, a '*mutual adaptation" perspecti^ (Berman and McLaughlin, 1975) was employed which 
recognized the evolving character of careet education and the diversity in concepts and approaches. 

Thirdly, previous research highlighted a number of forces which operated to facilitat a or impede 
the introduction of innovations into school systems. Career education was viewed as a bioad-ranging 
innoy-ation which would likely be affected by some of the same facilitating and impeding forces <call^ 



♦The OCE Monograph, A Primer for Career Education (Hoyt, 1977) provides a more detailed guide 
than was present when this study began. 
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determinants) which have been the focus of earlier studies. We reasoned that if local projects were un- 
successful in delivering career education to students, fault might rest with project inattention to some 
of the generic determinants of implementation. 

Lastly, we reasoned that projects are not themselves successful or unsuccessful; rather, com- 
ponents, strategies or activities are differentially functional or dysfunctional. Observing that target 
students outperform nontarget students on some outcome construct (e.g., career awareness) tells us 
nothing about why target students benefited or what can be done to improve the projects. 
Knowledge that some ill^efmed process is responsible for a desired outcome breeds false satisfacUon 
and confidence among the foolish, and frustration among the wise. RepUcation and exportation of a 
project require careful specification of process dimensions responsible for desired outcomes. 
Specifications of process-outcome relationships comprise.the heart of a good evaluation, and only such 
specifications provide a sound foundation for policy action. 

Exhibit 4 gives a schematic representation of the Implementation/Process Model. This model ex- 
pands upon the Conceptual Framework presented in Exhibit 2 (see Chapter 2). The model includes 
four compoments, three of which are featured in this chapter. Student outcomes and the relationships 
between exposure to career education and these outcomes are treated in the next chapter. The Im- 
plementation/Process Model posits that a set of determinants operates to create variation in the way 
career education is implement*^. Determinants are those factors which facilitate or impede any at- 
;«mpted educational innovation; these determinants may take on different forms depending upon 
specific characteristics of the innovation. 

Tiie process of career education manifests itself along essentially five dimensions: (1) changes in 
tangible resources (faciHties, staff, equipment, materials), (2) changes in intangible resources (staffing 
patterns, organizational alignments, student groupings). (3) changes in knowledge and undersUnding 
on the part of implementors (teachers, counselors, and administrators, etc.). (4) changes in implemen- 
tor role.'behavior, and (6) value internalization (i.e.. implementor commitment and attitudes toward 
the innovation). The level of change on these five dimensions is viewed as a product or outcome of the 
determinants. 

The quality of implementation interacts with "opportunity to learn" to determine level of ex- 
posure to treatment (i.e., career eduction) for individual students. Unlike curriculum innovations, 
career education does not presume some classroom-based instructional package. Thus, students 
within a classroom are not expected necessarily to receive identical treatments. For any of a wide 
range of reasons, including extroversion, likeabifity. or attendance, students can receive different 
amounts of exposure to treatment, even when the quality of implementation is uniform within a 
classroom or building. Similarly, even if "opportunity to learn" could be manipulated so that aU 
children received equal opportunity (e.g.. all students would have identical attendance, at a simplistic 
level), we would still expect the quality of implementation to vary ft-om teacher to teacher. Reliance on 
the individual students' level of exposure does not preclude the possibiUty that within-classroom 
variation may be smaller than between-classroom variation. The exposure concept presumes however, 
the possiblity of within-classroom variation. 

It should be not^ that the present study did not attempt to directly operationalize the construct 
"opportunity to learn." although it is included in the model for the sake of conceptual clarity and 
completeness. Due to the large samples of students involv«l in this study the costs involved in ac- 
curately measuring "opportunity to learn" were prohibitive. Thus, we simplified the model and 
assumed that the process dimensions dir«:tly determine students' "exposure to career education." 
which then cSontributes to determining learner outcome scores. We emphasize the absence of any 
measure of the "opportunity to learn" construct, primarily because it may account partially for^b-- 
served lack of congruence between student reported exposure and teacher reported implementation. 
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EXHIBIT 4 

A SCHEMATIC REPRESENTATION OF THE IMPLEMENTATION/PROCESS MODEL 



D«t«rminjint$ of implemsntation 



Political Complexity and External 
Agency Support 

Organizational Capacity and 
History of Innovation 

Congruence with Zeitgeist 

Opportunistic vs. Problem Solving 
Adoption Pfiilosophy 

Attitudes Toward Source 
of Innovation 

Adoptjon Strategies 

incentives and Disincentives for 
Adoption 

Explicitness 

Complexity 

Time for Planning & Orientation 

Resource Commitment 

Siturstion of Adopting 
Organisation 

Staff Characteristics 



Frocra Diminsk>ns 



• Tangible Resources 

• Intangible Resources 

• Implementor Knowledge 
and Understanding 

• Implementor Role/Behavjor 

• Implementoi' Value Internalization 



Exposure to CarMr Education 
Activities and Concspts 



Opportunity to Learn 



• SociabjHty 

• Teacher/Student Congruence 
on Sex and Race 

• Attendance, etc. 





Lsarnw Outcomtt 






Mathematics Achievement 




• 


Reading Achievement 




• 


Occupational Self -Awareness 




• 


Work Values 






Interest in Career Education 




• 


Attitudes Toward Work 




• 


Knowledge of Work Values 




• 


Decision Makmg 




• 


Job Seeking /Job Getting 




• 


Career Awareness 




• 


Economic Awareness 




• 


Academic Perceptions 




• 


Social/Personal Perceptions 



ERIC 



31 



32 



A QUALITATIVE EXAMINATION OF 
THIRTEEN DETERMINANTS OF IMPLEMENTATION 



This section brings together the results of onsite visits, extensive telephone discussions, and 
document reviews. The material to follow is more qualitative and impressionistic than that reported in 
the rest of the report. Although this information is not easily quantifiable, it is no less important for 
the reader to obtain a balanced view of the implementation/process of career education. 

' Determinants of Implementation 

Many factors operated to facilitateor impede the implementation of career education in the fifteen 
projects studied in this evaluation. Although the comments that follow are, strictly speaking, only 
applicable to the fifteen projects under study, the evaluation team has considerable confidence (based 
upon experience in evaluating numerous other career education projects) that the observations r^r- 
ted below can be generalized beyond our sample. The following comments are organized around thir- 
teen factors which operate to facilitate or impede implementation of career education projects. 

s 

Political Complexity and External Agency Support — Extent to which different interest 
groups and governmental agencies must be involved in implementing an innovation and 
the extent to which co^iizant Federal, state and local agencies and administrators express 
support and act in a supporting role. Also included is the status of relevant legislation, 
regulations, and policy pronouncements at aU governmental levels. 

OCE and other HEW units have demonstrated a high level of bureaucratic initiative in promoting 
career education. A complex of relationships and alliances have formed around career education, and 
predominantly positive attitudes exist between local projects and OCE. It is evident that local project 
staff feel a camaraderie with OCE. OCE has effectively employed face-to-face contacts between mem- 
bers of the bureaucracy andLE A, SEA, and community leaders, a technique which Litwak and Meyer 
(1966) suggest is a good indicant of high bureaucratic initiative. The Commissioner's National Con- 
ference on Career Education held in Houston in 1976. is another example of a bureaucratic initiative 
which had an observable, positive influence on the local projects. 

Although Federal assistance was largely a facilitating factor in implementing career education in 
the local projects, State day-Uniay involvement and support were absent in many cases, except those 
in which the project was a member of a statewide consortium. To what extent this low level of in- 
vojyement was due to direct OCE funding and monitoring, low SEA initiative, or adversary-based 
LEA-SEA relationships, is difficult to determine. In at least some cases, project staff felt no 
requirement to interact with respective SEAs, and the copy of the grant appUcation r«iuired to be 
sent to the State career education coordinator promoted little or no productive interaction. In other 
cases local personnel were much more enlightened regarding career education than were SEA person- 
nel and, as a consequence, had little incentive to seek guidance or support. Overall we must conclude 
that the LEA-SEA linkages were weak; nonetheless, perhaps because of the strong federal initiative, 
v/e observ«i no deleterious effects on implementation traceable to limited SEA involvement. 

Organizational Capacity and History, of Innovation — Extent to which the adopting 
agency has capacity to change; staff morale and esprit de corps among implementors; 
open communications channels; high level support for the innovation; previous successful 
experience with innovations; previous experience incorporating pilot or special project ef- 
forts into its ongoing operation. 

Most of the fifteen sites can be characterized as "early adopters" (cf. Rogers and Shoemaker. 
1971). Four sites have had ongoing career education programs since 1972, and only three sites did not 
have some ongoing career education effort prior to the 1976-77 school year. Nine of the local superin- 
tendents showed strong support for career education through soliciting local funds for carea- 
education activities and actively participating in awareness campaigns. Surprisingly, there was little 
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relationship between size of LEA and level of superintendent support. The highest levels of support 
were found in one of the largest ^nd one of the smallest LE A Perhaps the best indicator of superfn- 
tendent support is that seven of the nine sites seeking local monies successfully negotiated local fun- 
ding for the 1977-78 school year. 

Consistent with the observation that many sites are "early adopters'* was the favorable attitude 
toward innovation found in twelve sites. Only three sites were viewed as bearish on innovation, and, 
not so suprisingly, in two of these three sites superintendent support for career education was low to 
moderate. Foiir sites had major innovation efforts underway in addition to career education. At- 
tempts were being made to institute a new reading program, a new special education program* more 
instructional centers, and further implementation of state standards. None of these efforts was coor- 
dinated with career education activities. In general the fifteen sites had climates supportive of in- 
novation and superintendents who were highly supportive of career education. 

Morale of the local project staffs was unusually high in eleven of the fifteen projects. Although 
special project staffs are customarily zealous, most of the career education staffs 'were [particularly 
committed and had a high level of esprit de corps. An undoubtedly important contribution to observed 
high morale was the fact that six project directors reported directly to the superintendent, and seven 
others reported to highly supportive assistant superintendents. In at least three cases, the project 
director reported **out of line" to the superintendent, or assistant superintendent, giving the projects 
special recognition and blessing. 

Congruence with Zeitgeist— FiKtent to which the innovation is consistent with prevaiUng 
attitudes and social movements of lK)th a national and parochial character. 

Bureaucratic initiatives which are congruent with prevailing attitudes and social processes are 
more easily implemented than are bureaucratic initiatives which oppose such forces. At least at 
philosophical and conceptual levels, career education is in accord with the report^ needs and concerns 
program to promote full-scale implementation; demonstration, rather, was the primary objective, 
powerful social forces such as youth employment, the back to the basics movement, and nationwide 
reaffirmation of the importance of work. As presently configured, career education purports to em- 
body a generalized strategy of attack on some persistent social problems. Thus, from a national per- 
spective, career education appears to be congruent with the Zeitgeist and has not, as yet, provoked 
focused opposition from any powerful constituencies. 

Opportunistic vs. Problem Solving Philosophy— Extent to which the adopting 
agency is responding to the innovation from an opportunistic resource acquisition perspec- 
tive or a problem solving perspective emanating from locally defined and articulated needs. 

In seven sites the project director did not write the grant proposal, and in five of thes^ cases much 
idea generation was conducted by external agencies or individuals. Although we might take this as 
evidence of an opportunistic resource acquisition perspective, such a conclusion is unwarranted, given 
that four of these five sites have made long term commitments and are continuing projects under 
either local or state support. Although few of the sites conducted formal needs assessment studies 
prior to proposal submission, there is considerable evidence of coordinated proposal planning and very 
limited evidence of opportunism. 

Attitudes Toward Source of Innovation-- Attitudes of implementors toward the initiating 
agency or organization; attitudes toward the role of the initiating a^ncy as proselytizer and/ 
or enforcer; congruence between adopting agency and initiating agency perceptions of respec- 
tive roles. 

With few exceptions the projects viewed OCE's role as simply provider of funds. Only two proj- 
ects mentioned technical assistance as an OCE function, and only four projects mentioned OCE's 
monitoring responsibility. The fact that OCE was not considered as a source of technical assistance 
stems in part from the finding that only four projects were visited more than once by OCE staff. In 
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general, the attitudes of project staff toward OCE monitors were highly positive. The projects 
without prior career education experience wanted more OCE involvement, and it is our belief that im- 
plementation would have been facilitated in these newer projects had more technical assistance been 
forthcoming. 

Only three sites were satisfied with the grants process and the funding cycle. Several project 
directors reported that the protracted grant negotiations resulted in delays in implementation and 
that the one year funding cycle interfered with long range planning. OneVroject director commented 
upon the inconsistency between his intention to implement career education district-wide and the'con- 
straints imposed by year-to-year funding. It should be noted that Congress did not establish this 
program to promote full-scale implementation; demonstration, rather, was the primary objective. 

Adoption Strategies— Ways in which the implementors (e.g. teachers) are made aware of 
the innovation and instructed in its essential features: extent to which implementors are 
involvwJ in redefining emphases of the imiovation and making modifications to enhance local 
suitability and implemwitpr sense of ownership; inservice training (kinds and amounts). 

All projects employed teacher, counselor, and administrator inservice workshops as the primary 
adoption strategy. Projects varied considerably in frequency and duration of inservice sessions and in 
topics covered in such sessions. By far the strongest aspect of the general adoption strategy employed 
by the projects was the awareness component. A later section of this chapter will hi^ght the finding 
that most target teachers, counselors, and building administrators were aware of career education and 
were conversant with key themes. 

A clear relationship was evident between the extent to which projects delivered inservice follow- 
up and subsequent quality of implementation. This relationship emerged from on site observation and 
was strongly confirmed by the process questionnaire data. The number of inservice hours was 
positively related to number and .frequency of career education activities engaged in by teachers, 
counselors, and building administrators. The pacing and sequencing of inservice workshops 
throughout the year were important considerations in facihtating classroom-level implementation. 

Most of the projects adopted a top-down adoption philosophy, which provided Uttle opportunity 
for practitioner involvement in shaping the local conception of career education. One potential reason 
for this adherence to a top-down adoption approach was the finding that projects focused on 
maximizing "numbers of teachers inserviced" rather than maximizing intensity of inservice for in- 
dividual teachers. Laige numbers of teachers were inserviced. but fewer developed intensive relation- 
ships with project personnel. Intensity of inservice and instrumental assistance have been found in 
other studies to be important determinants of implementation or use (Louis, 1977). There was some 
evidence that project personnel were under pressures (whether internally or externally imposed) to in- 
service large numbers of teachers in keeping with a district wide implementation philosophy. Limited 
resources necessarily meant that many teachers received limited inservice: this w^ considered better 
than having a few teachers receiving extensive assistance. We must conclude that this emphasis on 
quantity rather than quality of involvement adversely influenced implementation in a number of 
projects. 

Incentives and Disincentives For Adoption— Type and level of incentives and disincen- 
tives to adopting agency, adopting units, and implementors including: money, publicity,"^ 
professional recognition, keeping up with the J oneses. 

All fifteen projects enjoyed positive publicity, and, in most cases, project directors received 
professional recognition for their efforts. The political leverage that often contes with grant money 
was used effectively by a few project directors, but many more wtjre unaware of how to convert theu- 
' financial initiative into resources to gain wider awareness and higher quality implementation. For 
example, the leverage technique of using OCE money to obtain long term local or state support was 
not practiced by more than a slight majority of the project directors. Overall, there was Uttle relation- 
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^ship between type of incentives or disincentives employed by a project and quality of implementation. 
What little relationship did exist could be explained by the superintendent's level of commitment to 
career education. 

ExpUcitness^Extent to which the theoretical basis for the innovation is. well articulated 
and the desired outcomes are well deOned and understood; extent to which the process 
activities (who, what, where, when, how) and alternatives are systematically delineated. 

4 

For the most part, the fifteen projects embraced all nine OCE learner outcbmes. This wholesale 
adoption strategy may be in itself indicative of the unfocused, unexplicit nature of many project 
^ philosophies and activities. Although some projects attempted to link specific process activities con- 
ceptually to specific learner outcomes, most of the linkages were tenuous. There existed, at times, an 
almost mystical belief that somehow career educatimt (in the abstract) if implemented, would result in 
a pervasive influence on all outcomes of interest. Project personnel were generally unable to catalog 
speciHc process' activities, or impose a conceptual framework on the processes. Discussions about 
process often developed into platitudes far removed from daily instructional decisions faced by 
teachers« counselors, and building administrators. 

If one factor stood out as an impediment to quality implementation, it was lack of process 
specification and attendant lack of delineated process-outcome relationships. If local project person- 
nel are to be left with the task of translating theory into practice, there needs to be some mechaxiism t^ 
assist them with conceptualization and implementation. Even prior to developing a technical 
assistance mechanism, more effort needs to be expended on conceptualizing the process of career 
education at the building and classroom levels. 

Comp/exify— Extent to which major adjustments in organization and role/behavior must 
be simultaneously implemented; number of people that must \^ involved in implementa* 
tion; number of external agencies and the respective levels of involvement considered desir- . 
able. 

The planning and implementation of a career education project fully consistent with the 
theoretical underpinnings of career education would be a complex endeavor indeed. This undertaking 
would require resocialization of actors ranging from teachers to business persons, involvement of 
diverse agencies not used to collaborating with one another, and alterations in fundamental beliefs 
toward education and work. Complex changes in^ teacher attitudes, roles, and behavior are not accom- 
plished easily. Gross, etal, (1971) and Evans and Schiller (1974) found that altering teaching 
strategies and role relationships proved much more difficult than changing administrative procedures 
or curriculum materials, and it is just such changes in roles and relationships that rest at the center of 
career education. It seems possible that, due to the complexity of the career education concept, the fif- 
teen projects found it difficult to be explicit; however, the wholesale adoption of outcomes and lack of 
clarity regarding process outcome relationships suggest that there are other reasons for specification 
failure. 

Time for Planning and Orientation— Amount of time to hire and inservice implementors; 
provisions for acquiring needed materials and facilities; and provisions for establishing 
necessary organizational linkages. 

There existed considerable variability in "time for planning and wientation" across the fifteen 
projects. In five cases project directors reported that too much was expected too fast. As might be an- 
ticipated, the projects with previous bacl^round in career education were able to start up quickly and, 
unlike many of the newer projects, had few initial implementation problems. Moreover, delays in the 
grant award process further impeded implementation, expecially for the less experienced project 
directors. 

Resource Commitment— Extent of dollars and in-kind contributions available for imple- 
mentation; per pupil or per unit expenditure. 
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Career education an extraordiiurily inexpensive innovation. Average per pupil expenditure 
rarely exceed nfty dollars and, in several cases, are les9 than ten doUar?. Wtoi viewed in light of per 
pupil expenditures for ESEA Title h ^ cmt of care^ education is alnKist inconsequential. Many 
Title I projects expend more than $400.00 per student (NIE, 1977) in the hopw of positively affecting 
a narrow set of learner outcomes. A natural question is how can career education be expected to affect 
not only basic skills but also career awareness and career knowledge when ESEA Title I has been only 
modestly successful with four to twenty times the resources available to mMt career education pro- 
jects? We are not prepared to answer this question in this evaluation, but we do think that 
observed treatment effects must be interpreted only after comparing the modest investo^t required 
to initiate a career education project with the resource available to other recent innovative 
programs * 

Saturation of Adopting Organization— Extent to which an a^ncy's units (e.g., schools) 
are involved in the innovation. 

Nine sites report that 100% of their elementary school practitioners have been at least partially 
inserviced on career education concepts and practices. Similarly, ten sites report complete saturation 
at the secondary level. We are inclined to interpret such reports as indicants of project emphasis on 
numbers as opposed to an accurate reflection of quality of implementation. Most of the projects have 
made a philosophical commitment to breadth of coverage rather than intensity and follow-up. Such a 
commitment would have been more defensible had projects specifically proposed objectives dealing 
with teacher behaviors and awareness. The focus on learner outcomes emphasized to varying degrees 
by most projects' proposals seems somewhat inconsistent with the projects' operational commitment 
to reaching as many teachers as possible. The doctrine of infusion may not be well served by short- 
term projects, and, furthermore, project emphasis on wide*scale implementation acts as an im- 
pediment to high quality implementation. OCE might consider discouraging such emphasis in short- 
term projects and instead encourage more of a demonstration orientation with careful attention to 
high quality implementation. 

Sfaff C/mmcf ens f ICS— Demographic characteristics of implementors including: age. sex. 
race, education level, past experiences, and attitudes toward change. 

Although we examine the effects of teacher, counselor and building administrator characteristics 
on implementation later in this chapter, it is informative to consider briefly several characteristics of 
the local project directors. Five project directors had little or no project management experience and 
two had no previous background in career education nrior to assuming mana^ment responsibilities 
on the career education project. Four other project directors had extensive project management ex- 
perience. All project directors had teaching experience, and ten also had buiIding-level administrative 
experience. We observed a high coincidence of low quality implementation and project director inex- 
perience, but given the large number of intervening factors, a direct casual relationship between the 
high coincidence of low quality implementation and project director inexperience ib unlikely. 

STUDENT REPORTED EXPOSURE TO CAREER EDUCATION 

Students were asked to indicate their frequency of participation in twenty-seven (twenty for fifth 
graders) activities. "The activities were organized into three sets: classroom activities, linkage ac- 
tivities, and extra school activities. In this section we summarize findings related to target-nontarget 
differences, grade level variation, project-to-project variation, and student background characteristics 
as determinants of student exposure. Rather late in the analysis phase of this study we discovered 



♦In the fifteen projects per pupil expenditures from project funds averaged from twenty to one hun- 
dred dollars. 
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that relationships Iwtween student reported exposure to carwr education and other variables such as.^ 
target-npntarget designation were conditioned by student ethnicity. Within^ as weU as across- 
projects, black students consistently report mor? frequent exposure to career Wue»tion activities 
than do white students. We are inclined to believe that this difference is due to overreporting by black 
students rather than true differences in exposure,* In an effort to account for this overreporting 
response we have analyzed the data separately for black and white students. 

Target-Nontarget Differences 

Among fifth grade students we observed a consistent pattern of differences indicating great|?r ex- 
posure to career education activities for target students. Consistent with expectations, classroofa ac- 
tivities are more discriminating at the fifth grade level than either linkage or extra school activities. 
Those activities which affect the greatest differences involve the use of media in teaching about 
careers, discussions about job values and interests, discussions of race and sex stereotyping, and par- 
ticipation in career field trips. The findings, although not overwhelmingly, ar6 clearly consistent with 
a conclusion that fifth grade target students are exposed to more career education-oriented activities 
than nontarget students. Although the overall pattern of differences between target and nontarget 
fifth grade students is essentially similar for blacks and whites, black fifth grade students consist* 
ently report more exposure to career education activities than white fifth grade students. Stated more 
precisely, there is about the same distance between black target and nontar^t students' regression 
lines as there is between white target and nontarget students' regression lines; however, the black 
students' regression lines are more elevated (reflecting a higher level of self-reprorted exposure) than 
white students. We may interpret these differences in at least two ways: first, we might conclude that 
black students do, in fact, receive more exposure to career education activities or, second, that black 
students tend to overreport. The weight cf evidence supports the later interpretation. 

, / Among eighth grade students target-nontarget differences are not as consistent in either direction 
or magnitude as those observed for fifth graders. Again we observed a marked tendency for black 
students, whether target or nontarget, to report more exposure to^ career education activities than 
white students. However, unlike the pattern of black-white differences among fifth graders, the eighth 
grad^ target-nontarget differences are not consistent for black and white students. Four of five 
significant dilfferences for black students favor target students, whereas only three of nine differences 
among white students favor target students. 

In general, both the direction and magnitude of tar^i't- nontarget differences are consistent for 
black and white eleventh graders. The majority of contrasts for black and white students favor target 
students although again black students report markedly higher levels of exposure. 

At the fifth and eleventh grades, there exists a clear pattern of differential expo.sure to carec^r 
education which is consistent with observed instructional emphasis at the twp grade levels: we ex- 
pected to find more exposure to classroom-based activities among fifth graders and more expo.sure to 
extra school activities among eleventh graders. The magnitude of target-ncn target differences in ex- 
posure is not overwhelming, but given the "numbers" oriented adoption strategy employed by most 
projects, it Is encouraging to find any process differences manifested at the student level In contrast, 
eighth grade findings reflect no consistent pattern of differential exposure favoring target students. 
The results for black students slightly favor target students, but among white students the advantage 
goes overwhelmingly to nontarget students. Several possible explanations for these discrepancies 
between fifth and eleventh grade results and eighth grade results are offered later in a section on 
teacher reported implementation. 



♦This is based on the similarity of results across and within projects and across and within grades 
where we have been able to compare the responses of black and white students. 
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Grade Level Variation 

V The magnitude of differences between fifth and eighth grade target students* reported exposure 
to career education activities vary dramatically depending upon whi^ther v^hite students or black 
students' self-reports are examined. If the combined results are used to evaluate differences between 
fifth and eighth grade target students' exposure, then one would conclude that fifth graders report 
more' exposure than eighth graders. Closer examination of these data, however, reveals that the 
relationship between grade level and self -reported exposure to career education activities is con- 
ditioned by whether the students are black or white. More precisely, there is an interaction effect. The 
vast majority of differences between black fifth and eighth grade target students favors the eighth 
graders, whereas just the opposite relationship holds for white students. Altijough the direction of the 
differences are opposite for black and white students, we should note that the magnitude of these dif- 
ferences is much larger among white students. Given what we believe to be a tendency for black 
students to overreport, we must conclude that fifth grade students perceive a higher level of exposure 
to career education activities than do eighth grade target students. 

Differences between eighth and eleventh grade target students follow an interesting pattern that 
does not overwhelmingly favor either grade level. Based upon the combined black and white students' 
results, there is a tendency toward higher exposure for eleventh graders. Eleven of twenty-seven con- 
trasts significantly favor eleventh grade target students, whereas seven contrasts favor eighth grade 
students. It is important to note that in only four of twenty-seven contrasts do black and whitp 
students differ in the sign of the correlation. Thus, for the most part, black and white students agree in 
their perceptions about which activities were more prevalent at the eighth and eleventh grades, but 
they disagree somewhat on their perceptions of the magnitudes of grade level differences, i.e., black 
and white eighth and eleventh grade students agree on the sig^n of the relationship but disagree on the- 
size of the relationship. Whether we focus on the sign or the magnitude of the relationship between 
grade and reported exposure among target students, we must be cautious regarding any inferences 
about the quality of implementation at the eighth and eh^venth grades when thgse inferences are 
based upon student reported exposure. 

Project Variai 

Overall, the within-project consistency across black arid white target students' reported exposure 
to career education is high. Note that we previously observed that black students tend to overreport, 
but, when we control for this tendency by analyzing the groups separately, similar patterns emerge for 
projects serving both black and white students. When black and white students are studied 
separately, much less discrimination is seen among the projects in black students' reportai exposure 
than white students* exposui In several projects which served both black and white students the 
combined analysis suggested chat these projects had significantly higher reported expdsure than 
other projects; however, when black and white student reports were analyzed separately these dif- 
ference disappeared. The cause of this anomaly was black students' tendency to overrepdrt, a ten- 
dency which is left uncontrolled in the combined analysis; any project with relatively large numbers of 
black students will appear to have higher bverp^l student exposure. When black and white students' 
reported exposure scores are analyzed separately, it is possible to find both gtoups actually are below 
their respective averagesu Thus, the question of which projects have higher student reported exposure 
to career education cannot be answered without knowing the ethnic characteristics of the students 
served in each project. \ 

In general, we mu^t be impressed by the regularity of pattern in student reported exposure to 
career education activities. The data are clearly not random in that those projects promoting Wgh ex- 
posure tend to do so on many of the activities, whereas projects which are l^ss successful in promoting 
student exposure tend to be less successful across a wide range of student activities. Similarly, the 
most successful projects tend to have success at all grade levels with both black and white students, 
and less successful projects tend to be less successful^at all grade levels with both b^cj^nd wKite 
students. Both the within-grade across-activity patterns and the across-grade patterns are stable and 
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highly Wggestive of differential jp^foject effects, although the magnitude of these effects generally 
not overwhelming. ^ . ^ 

Determinants of Student Reported Exposure 

There is no theoretically defensible reason for expecting student background characteristics to 
correlate with student reported exposure to career education activities. If, for example, males report 
less exposure to career education than females, then projects would need to investi^te iK)ssibIe 
reasons for this differential exposure since all projects profess to serve all students regardless of poat- 
secondary plans, sex, SES, or other characteristics. 

At all three grade levels, ^x, age, grade point average, and amount of television are essentially 
unrelated to student reported exposure. Amount of homework is consistently and positively related to 
classroom, liiUtoge, and extra school activities at all three grade levels. Ethnicity is the most con- 
sistent predictor, with black students reporting consistently higher exposure than whites. Overall, 
most of the correlations are small, suggesting that, with the exception of ethoicityt background 
characteristics are poor determinants of student reported exposure to career education. The fac£ that 
relationships between grade ai|d student, exposure, project and student exposure, and target* 
nontal^t group membership and* student exiK)sure are highly similar in direction suggests than an 
overreport response bias is operating in black students' reported exposure. Once this response bias is 
controlled by separately analyzing black and white students' responses many abnormalities in 
relationships between exposure and student outcomes disappear. 



TEACHER REPORTED IMPLEMENTATION OF CAREER EDUCATION 

A major premise of this study was that if the projects, have had' effects on educational prac* 
titioners, these effects should be manifested in increased knowledge and understanding of c^eer 
education; increased use of career education materials, concepts, principles and activities; and 
changed attitudes toward the role of career education in the ongoing instructional proems. In other 
words, the resocialization of educators should have resulted in changes in what they know, what they 
do, and how they feel. In this section we summarize findings related to target-nontarget differences, 
grade level variation, project-to-project variation, and determinants of teacher reported implement 
tation. 



Target^Nontarget Differences 

Among fifth grade teachers, the results supported a conclusion that target teachers (i.e., those 
identified by the local project staff as being actively involve in the career education project) engaged 
in career education-related activities more fi^uently than nontarget teachers, and that, apparently, 
inservice also had a positive effect^n knowledge and understanding and value internalization. Among 
eighth grade teachers the results were not as suggestive of a process effect as were the fifth grade fin- 
dings. Although there was a slight tendency for grade eight target teachers to be more active in 
role/behavior, this tendency did not extend to knowledge and understanding and value internalization. 
The results observed at the eleventh grade paralleled those at the fifth/ target eleventh grade teachers 
reported higher levels of knowledge and understanding, roleA>eK^^, and value internalization than 
their nontarget colleague. ' / 

Among fifth and eleventh grade teachers, we observe 9 pattern in the target-nontarget com- 
parisons of differential knowledge and /anderstanding, r^e/i^havior, and value internalization, 
suggesting that teacher inservice was producing a resociB^ng effect. In the comparisons of target 
and nontarget teachers, the larg^t ^process effects were found among fifth grade teachers, eleventh 
grade teachers were second, and eighth grade teachers were markedly less knowledgeaUe and active. 
It is perhaps no coincidence that student reported exposure to career educa^K^n followed a pattern 
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quite similar to teacher reported implementation, with eighth grade students reporting the lowest 
levels of exposure, and eighth grade teachers^porting the lowest level of implement5^tion. 

Grade I«vel Variation 

Differences between fifth and eighth grade target teachers were substantial Under knowledge 
and understapding, fifth grade target teachers repoi ted: (1) a fuUer understanding of career education, 
(2) more adequate preparation to implement career education, and (3) higher agreement with career 
education goals. Under role/behavior, sixteen of twenty-nine constrasts significantly discriminated 
between grade levels, with thirteen of these differences favoring fifth grade teachers. Under value in- 
ternalization, fifth grade target teachers perceived greater benefits in career education and believed 
more strongly than their eighth grade colleagues that a proper role of the schoolJis preparing students 
for the world of work. 

Fifth grade target teachers appeared to be considerably better than their eighth grade colleagues 
at imp:?menting career education activities. This finding held even when the fact that some activities 
are less appropriate at one grade level than another was considered. - 

There are substantial differences between eighth and eleventh grade target teachsrs' knowledge 
and understanding, roleAjehavior. and value internalization. Fourteen of thirty-nine ( ontrasts (36%) 
were significant, and thirteen of those contrasts favored eleventh grade target teachers. Eleventh 
grade target teachers .-eported having more information about whom to contact for career education 
speakers, more understanding of career education, and d higher perceived level of implementation 
than eighth grade target teachers. Eleventh grade target teachers engaged in a broad range of career 
education activities more frequently than eighth grade target teachers. We observed no differences on 
the three value intemaUzation constructs between eighth and eleventh grade target teachers. 

Project Variation 

Project variation in teacher reported implementation iclosely parallels project variation in stu- 
dents' reported exposure. Aggregating results across the three grade levels for white students, we find 
three out of five projects which are high on teacher reported implementation are also high on student 
reported exposure, with the other two evidencing mixed results. All three of the projects scoring low 
on teacher reported implementation also are low on student reported exposure. Two of seven projects 
which scored mixed on teacher reported implementation also scored mixed on student exposure, 
whereas three of the seven scored low, and two scored high on student exposure. Overall, this is a 
rather striking piece of confirmation for a process effect c.n teachers. Furthermore, this process effect 
appears to have translated into an effect on white students substantial enough for these students to 
report a difference in their educational experiences. 

~] . 

A similar pattern of relationships holds for teacher reported implementation and black students' 
reported exposure. Three of four projects with high teacher reported impl^ment^ion also evidence 
high student reported exposure. None of the projects that showed low tocher imported implemen- 
tation had enough black students to enter into the analysis. Of the foUr proje«lb r^^rting nuxed 
results for teacher implementation all four evidenced mixed results for black studtats' reported ex- 
posure to career education. Thus, again, we observe confirmation !or the conclusiWn that level of 
teacher reported implementation is related to level of student reported exposure to cl^eer education 
activities. 

Determinants of Teacher Reported Implementation 

The four sets of determinants of implementation (i.e.. staff characteristics, adoption strategy, 
teaching context, and organizational capacity) were related to variance in tocher reported implemen- 
taUon. The mo^ consistently predictable process measure was knowledge and understanding, with 
classroom activities, extra school activities, and linkage activities following close behind. The value 
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internaiization scales wvrv less pri»dictable, althouj^h in several cases the determinants explaintnl up- 
wards of twenty -five percent of total variance. It was heartening to observe that adoption strategy 
was consistently an important factor in explaining variance in implementation. The finding that 
"morale" was important at selected grade levels for selected prwess measures provides partial con- 
firmation of findings from the RAND change agent studies (Herman and McLaughlin, 1975). It ap- 
peariKi that the quality of implementation of carwr education is strongly dependent upon frt»quency 
and type of inservice education. Teachers receiving more inservice knew more about career education, 
implemented career education activities with niore frequency, and occasionally even manifestcni more 
positive attitudes toward career education's potential and ease of implementation. 

COiJNSELOR REPORTED IMPLEMENTATION OF CAREER EDUCATION 



Target-Nontarget Differences 

This section presents the differences between target and nontarget counselors' knowledge and 
understanding, role/behavior, and value internalization. We must caution the reader that the small 
nontarget sample obviates extensive generalizations about the counseling process and career 
education. 

The most striking finding presented by our results, exceeded only by the absence of many 
statistically significant contrasts, is the direction of the differences. Kxcept for two variables under 
knowledge and understanding (perceived level of implementation and agreement with carei^r 
education goals), only one other difference (efforts to reduce race and sex stereotyping) was 
statistically significant, and it favored nontarget counselors. Many other variables approached 
significanqib, and most of these favored nontarget counselors. One interpretation of these findings is 
that projects have done a poor job of inservicing counselors. Another interpretation is that the small 
sample of nontarget counselors is somehow atypical of the universe of nontarget counselors. An 
examination of the frequency with which target counselors engaged in selected career education ac- 
tivities suggested that the former interpretation has more credence. 

Project Variation 

The project difference*^, in counselor reported implementation bore little resemblance to project 
differences in teacher reported implementation. This finding, combined with the relatively low 
frcKjuency with which target counselors engaged in most career education activities, suggests that tht* 
projects realized a rather limited process effect on counselors. 

Determinants of Counnebr Reported Implementation 

The relationships betwtH^n tour sets of determinants and seven measures of counselor reported 
implementation of carei^r education were examined. The most interesting observation from the 
analysis was that adoption strategy was largely unimportant in predicting most of the process 
measures. Neither number of hours inservice nor type of inservice explained any of the variation in 
counselor reported implementation. Staff characteristics and counseling contest, on the other hand, 
were quite important. 

It is noteworthy that we observed few target nontarget differences among counselors and found 
that, among target counselors, amount and kind of inservice were unrelated to level of implemen- 
tation. This finding strengthened our conclusion that counselors were the weak link in the implemen- 
tation chain, and, apparently, a partial Explanation for low levels of counselor implementation could he 
traced to a lack of effei'tive inservice. 
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PRINCIPAL REPORTED IMPLEMENTATION OF CAREER EDUCATION 
Tai^et-No&targct EMfferencai 

The evaluation team found large differences between target and nontarget principals* knowledge 
and uftderstanding, role/behavior, and value internalization. The knowledge and understanding dif- 
ferences between target and nontar^t principals were substantially larger than those observed for 
the parallel teacher and counselor groups. Eleven of eighteen contrasts under role/'behavior were 
statistically significant, and all favored target principals. Under value internalization there were no 
significant differences between target and nontarget principals. 

Project Variation 

With the exception of two instances, most project differences appeared sporadic. Beyond these 
two projects there appeared to be little distinction in fmncipals' roleA>eimvii»' across projects. As we 
noted above, target princijmls engaged in career education activities more frequenUy than their non- 
target peers, but there was little project variation in the ways these target principals fulfilled their 
career education*related roli^. 

Determinants of Principal Reported Implementation 

Background characteristics of the principal, school context, and organizational capacity were 
generally inconsequential in determining target principals* implementation of career education. A 
complex of adoption strategy variables acted as the primary determinant of principal reported im- 
j)lementation. The major departure of these findings from those reported for teachers was that staff 
morale, as perceived by the principal, was not related to principal reported implementation. In con- 
trast, teacher perception of staff morale was very much related to level of teacher reported implsmen- 
tation; the higher the perceived morale, the higher the reported implementation. 

SUMMARY 

OF THE IMPLEMENTATION/PROCESS OF CAREER EDUCATION 




Aggregating reported levels of implementation for teachers, counselors, and principals, several 
generalizations about the implementation/process of career education are possible. Target teachers, 
counselors, and principals reported a high level of knowledge and understanding of career education. 
With near unanimity, target educators reported that the * 'concept of career education is^being actively 
implemented" in their schools. Similarly, all three groups strongly agree with the goals of career 
education/* Target teachers and principals feel they have a *'full understanding of th^sconcept of 
career education" and are "adequately prepared" for the implementation of career education., In terms 
of role/behavior, target teachers and principals reported substantially more frequent implementation 
of career education than nontarget teachers and principals. Target teachers enga^ in 
classroom-based and extra school activities most frequently, whereas prindpals were most often in- 
volved in linkage activities. Differences in value intemali2;ation between target and nontarget 
educators were infrequent. Fifth and eleventh grade target teachers showed some tendency to per- 
ceive more benefits in career education and less implementation burden than their nontarget 
colleagues, but these differences were not found between target and nontarget prindpals and coun- 
selors. Overall, we found educator knowledge and understanding of career education consistently af- 
fected by the projects, Role/ljehavior changes frequently were apparent, particularly among prindpals 
and fifth and eleventh grade teachers. Fewer changes in value internalization were apparent among 
members of the practitioner groups. 

Career education as implemented in the fifteen projects studied in this evaluation was associated 
with differences in educators' knowledge, reported profession^ behavior, and, to a lesser extent, at- 
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titud^. There is also evidence that the ways projects implement care<'r education and the setting in 
wiiich implementation takes place were related strangly to quality of implementation. Last, there is 
suppcut for the concluaion that the practitioners' changes were tran^ted effectively into xysw instruc- 
tional experiences for students. 

Among students at the fifth and eleventh grades, we observed a dear pattern of process effect 
favoring higher exposure for target students. We expected more target-nontarget dif^^ences in ex- 
posure to classroonvbas^ activities among fifth graders and more such difference in exposure to ex- 
tra school activiti^ among eleventh graders, and cmr expectations w^ a>nfim^. The observed 
process effects were not overwhelming, but they were sufHdently firequent and consistent in direction 
to conclude that fifth and eleventh grade target students had educational experiences which were dif- 
ferent from those of nontarget students. Grade variation consistently favored white fifth grade target 
students over white eighth grade target students in exposure to career education activities; however, 
black eighth grade target students reported higher exposure than black fifth grade target students. 
The differences between dghth and deventh grade target students were mixed Project-by^bject dif- 
ferences in student exposure evidenced a stable pattern. Projects promoting high student exposure 
did so consistently on a broad range of activities. Similarly, the mc^t succesusful projects tended to 
evidence success in promoting student exposwe at all grade levels, whereas projects having less suc- 
cess experienced difficulty at all grade levds. 
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CHAPTER IV 
THE OUTCOMES OF CAREER EDUCATION 



The primary objective of this study was not to evaluate career education, at least in the way the 
wotd ''evaluate" is used commonly. Although we present a number of findings that may be of interest 
to practitioners, our miain task was to assess tl^ relative effectiveness of evaluation models in ex- 
plaining changes in students' career education learner outcomes. In this regard we developed three 
complementary evaluation models; this chapter details the findings available from each. The models 
were developed to be complementary because of our distrust of results based on single 
methodologies. This distrust is shared by many evaluators and was expressed particiilarly well by 
Gilbert, Mosteller, and Tukey ( 1 974): 

Any attempt to infer what will be caused by an active intervention {''natural 
experiments'') or tmrandomized intervention has to be subject to possibiliUes of error that 
are hard to evaluate. Any attempt to assert that the statistical significance or confidence 
associated with such an analysis allows us to conclude reliably what active intervention 
will do is dangerous and unsound. 

The best we know how to do in such a situation is to seek out alternative methods of 
fallible inference, use up to several of them, and then, recognizing their fallibility, trust 
moderately in their combined message (p. 155456). 

We have applied our three models to the student data using as levels of analysis students, 
classrooms, schools, and projects. Each of these applications is detailed below, along with a discussion 
of the methodological difficulties, foreseen and not, inherent in efforts such as this. As will be clear to 
the reader, the results one obtains depend largely on the approach and attendant assumptions one ac- 
cepts. 

THE MODELS 

In order to meet the objective of identifying and testing a career education evaluation model, NTS 
developed three complementary models which could be used to explore the impact of career education 
processes on constructs measuring various student learner outcomes; these models have been 
described in Chapter 2, but will be reviewed briefly here. In applying these models it should be kept in 
mind that our main interest lay not so much with the statistical significance of an effect, but rather 
with the proportion of the outcome variance which was accounted for by an effect. 

Group Membership Model 

The first approach was the classical experimental design in which a nontarget group was Ui^ to 
estimate what the target students would have achieved had there been no career education program. 
The group membership effect, which we will also refer to as * 'treatment" effect, was estimated by the 
difference between the target and nontarget posttest construct means, adjusted for pretest differ- 
ences. The treatment effect can be re^pressed, however, as the proportion of outcome variance that 
can be explained by group membership (target vs. nontarget) after the differences in pretest perfor- 
mance have been removed. The higher the proportion^ the more we can be assured that membership in 
a target or nontarget group has had a bearing on student performance level m measured by the post- 
test measures. We have made the following distinction between outconle variance and criterion 
variance: outcome variance refers to the variance that is present in tl^ observed posttest scores; 
criterion variance refers to the residual variance that remains in the posttest scor^ after the variance 
that can be attributed to pretest variation has been removed 

Process Model 

The second approach assessed the relationship between students' exposure to career education 
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concepts and activities and their subsequent performance on the outcome constructs. If the career 
education program had an elfect, thcp we would expect that students with high exposure to career 
education activities would have obtained higher posttest scores (adjusted for pretest differences) than 
students with low exposure to career education. To the extent that exposure scores were able to ac- 
couct for outcome variance beyond that explained by pretest, we inferred a process effect. 



Joint Effects Model 

The third approach consolidated the group membership model and the process model into one 
analysis. In the joir*- effects model, we dealt only with criterion variance and examine4 what propor- 
tions of this varianj^ could be attributed to group membership and process components. Specifically,, 
we examined what proportion could be uniquely attributed to group membership and not to pro<»ss, 
what proportion could be uniquely attributed to process and not group membership, and what propor- 
tion was confounded between group membership and process. To the extent that the unique con- 
tribution of group membership to explaining criterion variation was large, we inferred that there was 
some treatment effect, but our exposure variables failed to specify the relevant aspects of the process. 
If, on the other hand, the unique contribution of exposiu^ variables to explaining criterion variance 
was large, we inferred that the program did little to affect outcome scores, but we were 
able to specify some of the important process dimensions which might be addressed by future career 
education programs interested in maximizing the particular outcome under consideration. Last, if the 
unique contributions of both group membership and exposure were small relative to their confounded 
contribution, then the program succe«ied in generating a treatment effect, and the evaluation team 
succeeded in specifying the particular process dimensions potentially responsible for that effect. 

CONSTRUCT DEFINITION 

Another of the major tasks of this study was a validation study of the learner outcome instrumen- 
tation. This substudy, which is discussed at length in Volume III, consisted of psychometric analyses 
of pretest learner outcome data, leading to the identification of outcome constructs. The procedures 
used for these analyses relied heavily on factor analysis. The items comprising constructs were scored 
using unit weights rather than factor weights to facilitate local usa^ of the findings. 

In addition to identifying nine learner outcome constructs within the career education-specific in- 
struments, we also developed two second-order factor scales from the Self-Observation Scales (a 
nationally normed self-concept measure), and generated compnrable reading and math scores from a 
wide variety of achievement test data made available to this study. Research by Stenner, et al.. (1978) 
has indicated, however, that the comparability of standardized tests, even though the tests may 
profess to be measuring the same areas of achievement, is questionable. Consequently, use of the 
achievement test data was limited to project-level analyses, and all plans to aggregate achievement 
test scores across projects and tests were abandoned. Exhibit 5 presents each of the learner outcome 
constructs employed in this study and the sources of items for each construct. In addition, we also 
analyzed the results based on the intact tests themselves and developed two "growth discriminant 
scores" (one affective and the other cognitive) based on items that showed the largest positive pre-posl 
change. A ' 'desire to work ' ' construct also was developed, based on item content validity. 
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EXHIBITS 



Learner Outcome Construct Measures'* 
and Item Sources 



Otttcfme 

Occupational Self- Awareness 
Work Values 

. Interrat in Career Preparation 
Attitudea Toward Work 
Knowledge of Work Values 
Decision Making 

Job Seeking/ Job Getting 

Career Awareness 
Economic Awareness 
Affective Growth Discriminant 

Cognitive Growth Discriminant 

Academic Perceptions 
Social/Personal Perceptions 

Reading Achievement 
Math Achievement 
Desire to Work 



S<mrce<8) of Items 
Ohio, Part 2 

New Mexico SOOl; New Mexico 5006 

New Mexico 5006 

New Mexico 5001 

PECE; Ohio, Part 1 

New Mexico 6002; New Mexico 5006; 

New Mexico 5007; New Mexico 500S 
Ohio, Part 1; PECE; New Mexico 5002; 

New Mexico 5006; New Mexico 5007 
Ohio, Part 1 
PECE 

Ohio, Part 2; PECE; New Mexico 5001; 

New Mexico 5006 
Ohio, Part 1; PECE; New Mexi«) 5001; 

New Mexico 5002; New Mexico 5006; 

New Mexico 5007; New Mexico 5008 
SOS-Teacher Affiliation and School 

Affiliatioii Scales 
SOS-Peer Affiliation, Self- Acceptance, 

Self-Assertion, Self-Security, 

and Social Confidence Scales 
Appropriate reading achievement 

test used in each project 
Appropriate math achievement t^t 

used in each project 
New Mexico 5001; Ohio, Part 2 



^With the exception of Interest in Career Preparation and Decision Making, these outrame meastu^s 
were employed at all three grade levels. 
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METHODOLOGICAL ISSUES 



Difficulties emerged in both the development of tt^ evaluation modela and the construct 
definition procedures. These methodological issues were anticipated by neither OCE nor NTS at the 
initiation of the contract Imt were discxivered during the course of this re^arch. Specifically, the 
following issues were identified and will be discussed below: adequacy of ccmstruct definitions, ap- 
propriateness of the aggre^tion level and appropriateness of the unit of analysis. 



Adequacy of Conatnict Definitions 

In this research nine ctireer education*specific learner outcome constructs were derived from the 
pretest data collected on several instruments. These instrumeiits were designed by their publishers to 
possess at least face validity in measuring one or more of the career education learner outcomes. Since 
face validity exists only in the eye of the beholder, and few consultants cmiid agree on what various 
items held in conunon, &ctor analysis techniques were used to explore ti^ structure of the data, and 
nine constructs huid thm compoi^t items were identified. These constructs aff<»rded a somewhat 
clearer refiection of the underlying structure; items which pc^sessed face validity but, in fact, only 
contributed **noise'' to the observations were weeded out. The rationale for the use of extensive 
psychometric analyses was that well-elaborated constructs should be more sensitive than the intact 
instruments to the effects of career education. Such sensitivity is needed as one typically finds, when 
pretest levels have been taken into accoimt, that less than h^f of the observed pmttest variation is 
left fur attribution to any treatment effect. 

The constructs remain, however, dependent upon the items that were bwrowed from extant in- 
struments. If these items prove largely insensitive to the effects of career education, then no amount 
of statistical manipulation will create a construct oh which treatment effects will be observable. Thus, 
while the individual items may possess face validity, and while sufficient interitem congruence may 
exist to permit the derivation of nine constructs, there is no guarantee that these items are sensitive to 
the changes that career education may produce. 

While a direct measure of such sensitivity could not be obtained, we were able to take an inferen- 
tial step backwards and assess indirectly the construct measure for such sensitivity by examining 
each construct's sensitivity to growth, Iwised on the pretest and posttest scores. Consequently, two 
additional outcome construct scores were developed that would be maximally sensitive to growth: the 
cognitive growth discriminant and the affective growth discriminant. These scores were computed 
from those items at each grade level that exceeded a minimal level of positive pre-post growth (as 
measured in standard deviation units). 

In addition to the discriminant scores, intact learner outcome total test scores were also used in 
the analyses. These instruments were developed with heavy emphasis on content validity. We argued 
that if the construct measures failed to indicate any effects, the discriminant and total test scores 
would be available to perhaps capture those effects and provide some sort of baseline against which 
the adequacy of the construct measures could be judged. 

Appropriateness of the Aggregation Level 

As originally conceived, the unit of analysis in this research was the student, and the level of 
aggregation was all students across ail projects. Two situations, however, made it advisable that 
other aggregation levels be considered. First was a problem regarding the student samples; many of 
the projects were unable to provide any nontarget student data. Thus, when the overall aggregation 
level was used, we were comparing two groups which differed at least in their geographic makeup, not 
to mention any other characteristics that may be unique to particular projects. This would suggest 
that if we are to apply the group membership model appropriately, it should at least be at the level of 
the individual project. 
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Second, and much more critical, was the problem of trying to eMmine a relationship, be it between 
outcome and group membership or outcqn^ and process, that may be inextricably tiai to the at* 
tributes of certain subgrwpa in the sample. In trying to examine tto relationship as it exists fc^ the 
aggregate of these subgrcmps, we may well have ended up with a cl(Hided in^age of the relationship as 
it exists in a subgroup, or worse, with a completely falK image. The implementation of carew 
education is by no means standard uniform. I^bstantial differences between the projects with 
respect to implementation have been found in this evaluation; it is possible, therefore, that what is oh- 
served in the overall results will basically be a hodge-podge which possesses no uniform structure and, 
consequently, r^sts the efforts of any ai the models to account for the variance. 

Appropriateness of Unit of Analysis 

In most research, the unit of analysis is taken to be the most elemental level of observation. For 
this research, the most elemental level was the individual student. It can be argued^ hawevec^ that 
sim^ the teacher is the vehicle for disseminating information, it would have been more appropriate to 
use the cbssroom as the unit of analysis. Using individual data in such a situation may have resulted 
in substantial proportions of outcome variation ibeing attributed to crrcr, consequently, a lesser 
proportion would be available for the examination of effects that could be attributed to group mem- 
bership or process. 

In order to assess this issue, analyses were conducted for all units of analysis under consideration. 
For the fifth grade these were student, classroom, and building, while for tae eighth and eleventh 
grades, these units were student and building. 

MODELS APPLIED TO OVERALL AGGREGATION 
OF INDIVIDUAL STUDENTS 

^^^..Data^^m all sites were pool«i for each grade level. The regression analyses necessitated, 
bbwever^ that each student in the analysis have no missing data. Thus, depending on the model and 
(he data it required, the number of available students and, consequently, the degrees of freedom 
varied considerably. 

There was a wide disparity between the target and nontarget sample sizes. Typically, the non* 
target group w.as from one-fourth to c e-tenth as large as the target group. In addition, the nontarget 
students were mostly drawn from only three projects at any grade level This raised considerable con- 
cern about the representativeness of the nontar^t sample. Application of the group membership 
model to the entire data set required the assumption that the nontarget students in three projects 
could provide a reasonable estimate of the no-treatment expectation for students across fifteen 
projects. 

These concerns generally were not eased by inspection of the target-nontarget pretest differences; 
the general pattern was for the two groups to differ by one-fifth to one-fourth of a standard deviation 
or more. Given pret^t differences of this magnitude, it was unlikely that the statistical adjustment of 
removing pretest from posttest would be adequate. The amount of growth shown from pretest to 
posttest was also unencouraging, particularly in light of the fact that fifth graders showed negative 
change on one of eleven constructs, eighth graders showed negative change on seven of thirteen con- 
structs, and eleventh graders showed negative change on seven of thirteen constructs. The propor- 
tions of the outcome variance for each construct that could be attributed to the pretest performance 
level were, in ^neral, moderate in sue« This indicated that while the measure were probably suf* 
fidently stable to be measuring the same constructs at pretest and posttest, there was still sufflcient 
variation left unexplained to permit treataient and/or process to acomnt for a sizable portion of the 
outcome variance. Although most outcome measure failed to register significant target-nontargi^t 
differences, there were some exceptions. Most notable was the Ohio Test, Part 1, which reflected 
treatment effects at all three grade levels. The construct based most directly on that test's items, 
''Career Awareness/* also reflected consistent but smaller treatment effects. The affective 
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discriminant showed a treatawnt effect at th« fifth said eighth grades, and "Attitudes Towards 
Work" aiHi the cognitive discriminant showed a treatment effect at the fifth grade; for these three 
measures, the ittoportiim of aigilainedcritericm variance langed from two to five pa^ ^ 

Results from all outcome measures converged on the conclusion that the evaluation of career 
education at the eleventh grade would be extremely difficult with a grwip membership model. To some 
extent the extremely small nontarget sample size can probably be blamed for the failure to find effects 
based on group membership. The poiail^ties will need to be considered, however, that either career 
education is not as effective at the eleventh grade, or. what is more likely, the instrumentation 
developed to assess learMr outcomes at Uie eleventh grade is insensitive to the effects that career 
education is exhibiting. 

The process model accounted for substantially more criterion variance than the group member- 
ship model for virtually all the leariMr cwtcbme measures at the overall aggr^tion level using the in- 
dividual student as the unit of analysis. For only a few of the measures was less than five percent of 
the criterion variani» being acoHmted for. 

What caused considerable concern, however, was that the effect of process on the outcome 
measure appeared to be far from uniformly pc^tive. Assuming that career education activities wcHild 
not have a deleterious effect, it is unclear what confoimding influences might be operating in this 
situation. For instance, one hypothesis might be that low scoring students tend to over-report their 
activity levels. What must be considered as well, however, are the concerns raised earlier in this chap- 
ter regarwiing level of aggregation and unit of analysis. Because these analyses used the individual 
student at the overall level of aggregation, a completely false picture of the relationship between 
proc^ and outcome may have been presented. 

Regardless of the reasons, when using individual students as the unit of analysis and the overall 
level of a^regation, process appears to have a generally negative relationship wiUi all the cognitive 
constructs and "Academic Perceptions" across aU three grades. Positive relationships do exist, on the 
other hand, with -some of the affective constructs and "Social/Personal Perceptions." It is very in- 
teresting to note that the constructs tended to be somewhat more sensitive to process effects than 
either the total test scores or the discriminant scores. The Ohio Test, Part 1, which was relatively sen- 
sitive to group membership effects, exhibited considerably less sensitivity to process. 

When the career education outcomes were assessed by the joint effects model, the proportions of 
outcome variance contributed by either joint effects of group membership models were minimal 
compared to the contribution made by process. In fact, for almost aU measures process accounted for 
over ninety percent of the criterion variance explained by the entire model. The only exceptions to this 
pattern were: the discriminant measures, the "Career Awareness" construct, Ohio Test. Part 1. at 
grade five; both parts of the Ohio Test and the affective discriminant at grade eight: and "Career 
Awareness" and Ohio Test. Part 1. at grade eleven. This indicated that fhe cognitive items of the Ohio 
Test are apparenUy reflecting an outcome that is sensitive to group membership but not to the 
process activities assessed in this study. 

In general, the joint effects model provided no additional insights over and above those gained by 
the other two models. This was reflected clearly by the virtually nonexistent proportions of criterion 
variance attributable to the joint effect. The group membership and process components werp 
operating virtually independenUy of one another. The implication of this finding is that the activity 
variables that measure pKX^ss are not sensitive to those differences in process that would be caused 
by membership in either the target or nontarget group. For all intents and purposes, group member- 
ship and process are accounting for disjoint portions of criterion variance, with the portion accounted 
for by process being substantially larger. In other words, if the educational experiences of target 
students vary from those of nontarget students, the experimental differences that produced the effect 
are not being reflected in the activities. On the other hand, the activities are measuring some part of 
the students' experiences .that are reflected also in the learner mitcome measures. The group member- 
ship and process models, therefore, seem to be assessing separate portions of the students' 
experiences. 
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Our analysis of mdividusl student data at the overall level of aggregation yiekkd three maja* con* 
elusions. Firstt the gnmp membership data indicated that the treatment effect was minimal, if not 
nonexistent; nonetheless, where the inttfi^tion of i^pup mraibership and pntmt was small the 
treatment effect ten^d to be positive the fifth and eighth gracbs. Second, the pnxms model in* 
dicated a moderate to strong rdaticmship between {mx^ss and outoune. The relationship was almost 
exclusively negative fw the cognitive constnicts, howe\w« and mued for the affecti>^ constructs. 
Thirds the joint effects model indicated that thid effects of process and treats^t cm outcome were vir- 
tually disjoint, with process accounting for the bulk of tl^ variance explaii^ by ti^ joint effects 
model; that is, the activities chosen to measure tl^ process of career education were not sensitive to 
those aspects of membership in a career education project that affect performance on outcome. 

The poor showing of the group membership model raised strong concerns about the ap- 
propriateness of the nontarget group sample as a source of no-tr^tment expectations. It was 
therefore necessary to consider wfaetiter it would be more appropriate to use the nontarget groups only 
to generate no-treatn^t expectations for the target groups in their sites, that is, to change the level 
of aggregation from the overall to the project leveL 

The apparently negative relationship between process and outcome also raised strong concerns. It 
was recognized that insufficient information existed for exploration of the existence of some subtle 
underlying influence that could result in the negative relationships. The possibility of the negative 
relationships being an artifact of level of aggregation was available, however. In other words, it was 
possible to investigate whether subgroups in the sample could actually have a positive relationship, 
even though, due to their pre- and posttest means, aggregation of their data yielded an apparently 
negative relationship. 

Given the unique nature of the sites included in this study, it does not seem at all unreasonable, in 
the light of the proceeding discussion, to question whether the assessment of career education can 
truly be accomplished at a national group level. The implementation of career education is by no 
means standard or uniform; substantial differences between the projects with respect to implemen* 
tation were highlighted in Chapter 3. 

For all of these reasons, it seemed appropriate to lower the level of aggregation from the overall to 
project level, while retaining the student as the unit of analysis. Making this change presumably 
would yield more appropriate target-nontarget comparisons and, perhaps, reveal the presence of 
positive process-outcome relationships. 

MODELS APPLIED TO PROJECT AGGREGATION 
OF INDIVIDUAL STUDENTS 

At each of the three grade levels there were three projects with a sufHcient number of both target 
and nontarget students to allow the application of each of the three evaluation models. It should be 
emphasized that analysis of the results at the project level was never envisioned as being able to 
produce any sort of nationally generali^ble results. In light of the aggregation discussion such a goal 
would be inappropriate. Rather, the objective was to see if, by appljing the models at the local level, 
the models would be able to account for a substantial portion of the CHitcome varian^^ and, thus, be 
able to describe how career education operates at the project level. If there had been sufficient sub* 
jects, it probably also would have been appropriate to further divide tl^ projecte into racial and sex 
subgroups to assess the effects of grouping and group context on the outcome-proo^ andmitccmie- 
group membership relationships. This was not possible. Since the data wet^ bein,; analyzed at the 
project levels it was possible to include the achievement test data because the same instran^t was 
used within* although not across projects. It was still necessary to exclude some of the learner out* 
come, achievement, and SOS measures from the analyses, however, due to the limitations forced by 
small numbers of students. 

Analysis of individual student data at the project level of aggregation jdelded some intriguing 
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restilts icross the three grade levels. The group membership model tended to reveal the 
pr^ence of treatment effects a ; the local level that were no more -sutstantial than the tseatment 
effects observed in the overall analyses. Only one of the site-grade sets consisttotlstahowed substan- 
tial positive treatment effects; fw the rest of the site-grade combinations, treatment effect* were 
either nonexistent or confounded with interaction effects. 

The process model at the project level of aggregation tended to reveal much more positive 
relationships between process and outcome that were observed at the overall level. Several sites 
exhibited generally positive relationships between process activities and virtually all the constructs at 
each grade level. 

The joint effects model continued to show few, if any. effects that could be attributed to the joint 
effects of process and group membership. The effects of process and group membership tended to be 
disjoint across all the sites, with the bulk of the explanatory power of the jflint effects mpdel vested in 
the process component. The constructs tended to be more sensitive to tm process and treatment 
effects than the total test scores. Although the Ohio Test. Part 1, provided an occasional exception to 
this pattern, the effects were not as strong as those observe in the overall analyses. 

Thus, even at the project level of analysis, either an appropriate nontarget group could not be 
found, or the outcome measures were not sufficiently sensitive to effects that could be attributed to 
group membership. The emergence of strong process-outcome relationships at the project level, 
however, provided a verj' strong indication that level of aggregation could be & crucial determinant of 
the observed polarity of the relationship. While, admittedly, the polarity at the project level was by no 
means uniform, its shifting quality indicated that site characteristics might have a much stronger in- 
Huence on the process model than was previously suspected. 

The inability of the group membership model to work left us in a quandry. If our measures were, in 
fact, insensitive to group membership, it seemed contradictory that they should be so sensitive to 
process. The noncomparability of the target and nontarget groups seemed a likely explanation, but 
this was a post-hoc conclusion which provided lijpited guidance for further analyses. Conferences with 
staff at OCE led to the recommendation that a level of analysis higher than the student be used. 
Career education, it was pointed out, is delivered at the classroom level. For that matter, Herman and 
McLaughlin (1975) found that car^r education was implemented most effectively when principals 
were supportive. This suggested tUt the classroom or the building might be used as the unit of 
analysis. If the effects were occurrng at the classroom and/or building levels, then using the student 
as the unit of analysis might have resulted in so much extraneous variance that the presence of main 
effects was obscured. 

MODELS APPLIED TO OVERALL AGGREGATION 
OF BUILDING AND CLASSROOM DATA 

Due to the nature of the educational environment at the eighth and eleventh grades, students 
cannot be identified as belonging to any one particular classroom. They circtilate, mstead, through a 
variety of classroom situations, some, none, or all of which may provide exposure to career education. 
Consequently, only the building could be used as the unit of analysis at these grades. The building and 
classroom were both used as unite of analysis at the fifth grade, however. 

Using the building as the unit of analysis did not enhance the explanatory power of the group 
membership model, nor did changing the unit of analysis substantially alter the direction of the 
treatment effect; at the fifth and eighth grades the group membership effect was generally positive, 
while at the elevehth grade the effect was generally negative. - 

What was especially interesting, however, were the results of the process model, although it only 
could be appUed to the fifth grade data. I^s than six percent of the'key activity variable relationships 
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with outcomes were negative; that is, the relationships between process activities and outcomes ^ 
positive across nepiy all the learner outcomes. The other results from the process model were 
generally discc^pMd due to the low degrees of freedom a^tl^ large number of model parameters. 
The joint effeete model continued to show little in the way of joint effects; the process component once 
again constituted the bulk of the model's explanatory [^wer. 

The models also were applied to the grade five data using the classroom as the unit of analysis. 
For the group membership model, the treatment effects continued to be small but predominantly ^ 
positive. * 'Attitudes Towards Work" and •'Career Awareness'' exhibited somewhat more sensitivity^ 
to group membership than was evident in the overall analyses of the Hfth grade student data. The 
process model revealed a strong positive relationship between process activities and outcomes. W^hile 
the relationships were not as positive as those observed in the analyses of the fifth grade building4evfcl 
data, they were considerably more positive than the overall student analyses had indicated/In ad-^ 
dition, the relationships were consistently positive across all the learner outcome constructs. 

In this section, analysis of building and classroom data at the overall level of aggregation yielded 
three conclusions. First, the group membership model did not reveal the presence of any treatment ef- 
fects that analyses using student level data had not indicated. Second, the process model revealed,^ 
least at the fifth grade level, the emergence of positive process-outcome relationships when a unit o?^ 
analysis higher than the student level was used. Furthermore, the higher the level of analysis unit 
utilized, the more positive were the relationships. Third, the joint effects model did not reveal the 
presence of any component effects that had not been observed through analyses using student level 
data. 

At this point, four different combinations of aggregation levels and units of analysis had been 
analyze using the three evalij(ation models. The results from the group membership model did not 
vary markedly from one combination to another; the effects tended to be small at all grade levels and 
were g**nerally negative at grade eleven. This pattern suggests two explanations. First, while the 
process activities and outcome variables ^ye stronjg relationships with one another, the processes 
have not captured whut truly occurs in a ^f^erally-funded career education project, nor do those 
processes reflect the project's consequences. Given that the instrumentation used in this study was 
based on off-the-shelf products, oftep developed for a single project, this fM)Ssibility cannot be 
dismissed. Fui*lher, since the construct measures proved insensitive to growth, it is important that 
any future evaluations of career education be preceeded by the development of new process and out- 
come instrumentation that will possess sensitivity to the effects of growth and treatment. 

A second explanation for the lack of a group membership^ffect is the noncomparability of the 
target and nontarget student groups. Suppose, for example, that the process of career education goes 
on, regardless of the presence or absence of formal federal funding. The federal program might be .sup- 
planting an effort that would otherwise betaken on by the school administration. The relatively small 
expenditures necessary for career education reinforce the potential for this tendency. A related 
possibility might he that federally-funded career education programs have an effect on the entire 
cohort m which the target group is embedded. For example, nontarget teachers could convey to their 
students information acquired from interactions with target teachers. It has been frequently observed 
in compensator^' education programs that nontarget students benefit by being in a target school; 
materials and the effects of teacher inservice training will not likely be confined just to the target 
population. That target and nontarget teachers' reported levels of implementation were not vastly dif- 
ferent suggests that this may occur. 

These possibilities, along with the strong process-outcome relationships observed throughoutr 
this study, have led us to conclude that the process model will protebly be the most useful for 
evaluation of career education programs. It is a model that must be applied with great care, however. 
As has been evidenced time and time again throughout this study, the effects of aggregation level and 
unit of analysis on the nature of the observed relationships can be profound. We have seen that the 
overall aggregation of student level data tends to yield negative rektionsh^s, but as the unit of 
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analysis is raised from the student to the classroom and building levels the relationship becomes 
strongly positive. The relationships can be positive, Mgative, or mixed wi^ student data are 
aggregated to the project level. It seems clear, then, that the prin^uy source of this instability is tied 
to the characteristics of the individual students. It was deemed aiipropriate, therefw*. that this chap- 
ter sheuld conclude with a somewhat more detailed application of the process model, both from the 
'viewpoint of assessing how transitory the process-outcome relationships are and also providing the 
reader with some insights into the relationships between individual activiti^ and outcomes. 

PROCESS MODEL APPLIED TO VARYIXfi AGGREGATION'S ' 

OF INDIVIDUAL STUDENTS 

The purpose of this section is to apply the process mo<iel to various aggregations of student data 
to examine the sensitivity of the process-outcome relationships to the nature of the a^regation. In 
considering what sorts of a^;regation would perhaps be most sensitive, there were two logical choices. 
The first was to aggregate by project, the effects of which have been documented earlier in this chap- 
ter. The unique characteristics of the sdtes, such as level of imidementation and demographic dif- 
ferences, ^med likely to {adduce differing proc^ss-outoime relationshijra. The second choice was to 
aggregate by ethnicity or, more specifically, by black and white students (other ethnic groups were not 
represented in sufficiently large numbers). Chapter 3 document^ that blacks tended to report more 
exposure than whites, much as students in one project tended to report more exposure than students 
in other projects. While the combination of the two analyses into one was considered, the absence of 
either sufficient blacks or sufficient whites within any one of the projects precluded this alternative. 
Further, to eliminate one possible source of confounding, only target students were included in these 
aggregations. ^ . , . 

Analysis of target student data using the process model at project and ethnic aggregation levels 
yielded three primary conclusions. First, certain activities possess high correlations with outcome 
measures across all projects and ethnic groupings. The following activities were consistently and 
positively related to virtually all the learner oucomes at all grade levels: talking with other students 
about jobs and careers, listening to their own parents talk about their jobs and careers, asking their 
own parents about jobs and careers, and taking part in activities designed to reduce race and sex oc- 
cupational stereotyping. At the eighth and eleventh grade levels three additional activities tended to 
be positively and consistently related to the outcome measures; these included: studying the interests 
and abilities needed for particular jobs and careers, learning basic skills through the infusion of career- 
relevant infection, and learning about their own work values. There were also two activities that 
presented consistently negative relationships across the outcomes at all three grade levels: 
simulations of world of work experiences in the classroom and listening to other parents make presen- 
tations about their jobs and careers. At both the fifth and eighth grade levels', attending career fairs 
and taking home career education materials also were related negatively to most of the outcomes. 
Other specific relationships are detailed in Chapter 6, Volume 11. 

Second, the pattern of correlations between activities and outcomes across project and ethnic 
groupings can vary substantially; the scope of this study was not sufficiently broad, however, for us to 
begin to identify the determinants of such variation. 

Third, some learner outcome measures tended to be more sensiUve across grades to exposure to 
the acUvitiea than were other outcomes. Occupational self-awareness, work values, and sociai/per- 
sonal perceptions tended to be positively related' to the activities at aU grade!, and interest in career 
preparation was positively related to the process acUvities at applicable grade levels {i.e., eighth and 
eleventh) At the same Ume, the relationships between some of the learner outcome measures and the 
activities tended to be negative. The only outcome faUing into this category across ail three grades 
■ was math achievement. 

In general, while the process model Would appear to be the proc«lure of choice for evaluating 
career education projects, considerable care must be taken in its application. While the impact of cer- 
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tain activities on outcomes seems to be unaffected by changes in grouping, nonetheless/ such 
activities are clearly in the minority. Substantial 6il ^rences exist between projects and ethnic groups 
regarding the nature of the proe^-outcome relationships. In the absence of any additional infor* 
mation at this time that would serve to explain BndJor predict the occurrence of Uuese variations, the 
career education w^aluator would be well advised to implement process models on groups that are as 
homogeneous as possible. While the extent to which the project participants can be subdivided intx> 
homogeneous groups is certainly limited by administrative necessities, a few gross groupings based 
on ethnicity and general background would be well recommendeid' 

SUMMARY OF THE OUTCOMES OF CAREER EDUCATION 



y In this chapter we reported the results based on the application of the group membership, process, 
and^joint effects evaluation models to the student learner outcome data collected daring the course of 
Sliis study. From the many particular findings ba^ on these applications, three general conclusions 
^st^d out. First, the learner outcome construct measures used in this study did not tend to be sen- 
4duve to growth. Although the process model was more effective than the other models in explaining 
pre-post change, the fact remains that there was little change to be explained. We are inclined to inter* 
pret this more as a problem in measurement of the outcomes than as an indication of a lack of real 
growth. 

Second, the clients of career education evaiuators would be better served through the application 
of thp^rocess than the group membership model Probably the only instances in which the group 
mei^bership model will provide useful information are those rare cases where, by random assignment 
or fortuitous circumstances, the treatment and control groups are similar on demographics (par- 
ticularly ethnicity) and pretest scores. Even if those conditions can be met, however, we still are in- 
clined toward the process model as it provides detailed information not only about overall results but 
also specific process-outcome relationships. 

I'hird, the results one obtains from the applicatior of the models are conditioned by basic choices 
of the unit of analysis and level of aggregation. In this study we not only observed the magnitude of 
effe^^ts to be dependent on the unit of analysis and/or level of aggregation used, but even found 
changes in the direction of those effects as the units of analys e and levels of aggregation were 
manipulated. 

As has been noted frequently in this report, this study was neither designed nor conducted as an 
evaluation of career education. Nonetheless, i is clear from the results of our tests of the three models 
that both the supporters and the critics of career education could use selected results to buttress their 
convictions. We strongly urge that the results presented in this study not be used in this way, but 
rather that these results be used to help further clarify the career education concept and facilitate its 
implementation. Only when these steps have been done will evaluations of career education itself be 
useful ^ 
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CHAPTER V 
ANALYSIS OF KEY POLICY 
ISSUES AND RECOMMENDATIONS 

Much has been learned about career education during the twenty-four months of this study. Some 
of our findings confirm common sense expectations; others provide new insights into the implemen- 
tation/process of career education and its effects on students. This chapter organizes selected findings 
around key policy issues facing career education planners at local, state, and federal levels, 

GUIDELINES FOR GRANT PROPOSALS 

The OCE guidelines for grant proposals should be redrafted to foster more precise thinking from 
prospective grantees. It is not sufficient, however, to simply request that grantees be more specific; 
rather, a proposal format is needed that will provoke grantees to address certain definite questions. (1) 
What are the proposed process changes that the project hopes to attain? For example; what changes 
in tangible resources, intangible resource, implementor knowledge and understanding, implementor 
role/behavior, and value internalization are the projects trsdng to accomplish? (2) What adoption 
strategies are proposed; how much inservice is done on what topics and with what frequency? (The 
present evaluation study has documented adoption strategy as the single most important deter- 
minant of implementation). (3) What are the project level detenninants of implementation which 
might affect the proposed project, and how will these be manipulated to facilitate implementation? (4) 
Apart from wholesale adoption of all nine OE learner outcomes (which, incidentally, should not be en- 
couraged), what are the most important outcomes that the project is working toward? (5) How do the 
^process dimensions relate to the learner outcomes, i.e., what ar^ the linkages between proposed 
changes in student experiences and subsequent ciianges in student's betiaviors and attitudes? 

We wish to emphasize two things about proposed changes in the guidelines. First, to anticipate a 
rejoinder^ there is nothing in the proposed changes that erodes OCE's nnnprescriptive and 
collaborative posture toward the field. We are not proposing that OCE present e certain implemen- 
tation processes or require a particular adoption strateg;^; rather, we perceive the need for a generic 
conceptual framework which will facilitate more precise planning and higher quality implementation. 
Second^ it would be insufficient to simply list a set cf questions sintiilar to those presented earlier in the 
guidelines. Prospective grantees need to understand how the conceptual framework is structured, so 
that it can become a useful tool rather than another bureaucratic hurdle. 

Last, we do not pretend to have formulated the ultimate conceptual framework for viewing the 
implementation/process of career education. Much more could be learned from the expanding 
literature on implementation that would improve the framework and provide it with a firmer 
theoretical foundation. Our efforts in this study represent only a beginning, but even this modest 
beginning holds promise for dramatically improving the implementation/process of career education. 

LEARNER OUTCOME INSTRUMENTATION 

Throughout the life of this study, we have bemoaned repeatedly the non-existence of good learner 
outcome instrumentation. Projects are gravitating toward learner outcomes for which adequate in- 
strumentation is available; thus, a severe mismatch is developing between what is being taught and 
what is being tested. Both the short and long-range consequences of this schism are dangerous. In the 
short run, projects will b^ unable to muster more than weak treatment effects without qu^ty in- 
strumentation; in the long run, career education will be unable to launch meaningful longitildinal 
studies due to the inadequate definitions appended to most career education outcome constructs. Fur- 
ther, available instrumentation may tend to dictate instructional emphases; this is quite different 
than instructional needs dictating the instruments. OCE and/or NIE should seriously consider 
funding a comprehensive instrument development effort with provisions for specifying in more detail 
the range of constructs considered important in career education. Such a study might give particular 
attention to the intrinsic and extrinsic definitional properties of proposed construct measures. 




OCE is faced with a catch 22 situation, as far as funding an instrument development study is con- 
cerned. Commercial test publishing companies are not going to move swiftly into career education un- 
til there is convincing evidence of its long-term viability. Career education's continued acceptance 
depends, in part, upon well-implemented evahiations documenting its benefits; however, good 
evaluations remain a vain hope without quality measiu^s of commonly agreed upon constructs. It has 
taken the Office of Child Development ten years to reali2e that the early chUdhood market is not at- 
tractive enough for publishers to expend the resources required to develop an appropriate early 
childhood battery. Early childhood evaluations, and policymakihg based upon these evaluations, have 
suffered accordingly. It is unlikely that career education can afford to wait ten years before embarking 
on the kind of instrument development recommended here. 



In spite of frequent pessimistic notes about the lack of measurable process effects of many in- 
novations, we submit that process delivery of career education is possible and was accomplished to a 
significant degree in the fifteen projects. We also are persuaded, however, that the process was poorly 
specified in most projects and, had process specification been more precise, implementation would 
have been better. 

The conceptual framework developed for this evaluation operationalized the implemen- 
tation/process of career education as seven constructs: knowledge and understanding, classroom or 
counseling activities, linkage activities, extra school activities, perceived benefits of career education, 
perceived burden of career education, and perceived role of school in implementing career education. 
This formulation did prove valuable for evaluating career education but, more importantly, it also 
demonstrated the importance of conceptualizing and precisely specifying desired career education 
changes in educator s knowledge and understanding, roleA>ehavior, and value internalization. The 
potential benefits of a well-articulated schema for specifying the car^r education process to present 
and future career education efforts are substantial. 

It is recommended that OCE prepare a generic framework far classifying alternative career 
education processes and field test the framework on a representative sample of career education 
efforts. One possible difficulty with such a developmental effort can be anticipated. Although we now 
know a great deal about what factors facilitate and impede implementation of a career education 
project, we do not know the extent to which these same factors are important for attempts to in- 
stitutionalize career education-based curricula throughout a school district. It is our suspicion that 
most of the determinants of implementation examined in this study would apply to such a large-scale 
effort, but some other factors also would undoubtedly come into play. Since we did not study this in- 
stitutionalization process in our evaluation, we are unable to comment on the nature of some of these 
additional factors. It is clear, however, that infusion and institutionalization are not well-understood 
concepts; project staff often confused them with aggregated counts of inserviced staff or numbers of 
buildings with career education coordinators. We recommend that OCE examine further the meaning 
of infusion and institutionalization and delineate what it means for a project to move toward these 
goals. 



Many of the projects studied in this evaluation could have made excellent use of technical 
assistance in the early stages. The inexperienced project di. ectors, in particular^ had little opportunity 
to learn either the rudiments of project management or. more importantly, how to manipulate the 
determinants of implementation to facilitate quality implementation. We observed that some projects 
suffered short-term credibility problems within the school district because the project 4irector was 
preoccupied with learning the mechanics of project management. Some %ell-informed technical 
assistance during these early stages could have improved implementation measureably. Among the 
topics in which technical assistance would prove especially beneficial are the following: (1) theory and 
application of adoption strategies, (2) day-to-day management of a training and development effort, (3) 
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planning and budgeting, (4) politics of special purpose projects, (5) conceptualizing the process of 
career education, and 16) general career education instructional techniques and methods for involving 
the total community in career education. We recommend that OCE consider alternative mechanisms 
for providing technical assistance, with the full recognition that OCE currently does not have suf- 
ficient inrhouse personnel to supply the amounts of assistance required by many newly /umii?d 
projects. 

The present study combined providing evaluation technical ass^istance to the projects with im- 
plementing fifteen full-blown local evaluations and one overall, or common, evaluation. Mixing 
technical assistance and onsite operational responsibilities resulted in neither service being effectively 
provided. Most projects, knowing that NTS was responsible for their local evaluations, did not 
develop a deep commitment to the total evaluation efforts, NTS, for its part, was so occupied with 
managing a large-scale, multi-site evaluation that, beyond initial design conferences, no intensive on- 
sive technical assistance was provided, // technical assistance is to be effective^ operational respon- 
sibilities must reside uncompromisingly with the recipient of that assistance. If large-scale evaluation 
is to be successfully implemented, operational responsibilities (particularly instrument ad- 
ministration) must be the exclusive province of the contractor. 

PROJECT FUNDING MECHANISM 

Projects operating under external funding are marginal enterprises appended to ongoing systems. 
The exernal funding usually leads to underdeveloped, hectically planned programs. Thus, the wide- 
scale implementation strategy leading to program institutionalization embraced by most career 
education projects is not well-served by short-term, external funding. 

The project mechanism is at its best when the primary objective is demonstration, documen- 
tation, or awareness-building. When the goals shift to institutionalizing a concept, then the project 
mechanism is not appropriate, and completely different strategies are required. 

Project directors often confused the notion of institutionalization with inservicing large numbers 
of staff. Such importar* long-term issues as locating ongoing responsibility for career education 
within the organization and gaining local support for career education were not associated with im- 
plementation in the minds of most project directors. We have observed several times that projects 
should be discouraged from playing the **numbers game" to enhance implementation of career 
education ideas and, instead, should concentrate intense inservice activity on a manageable number 
of staff. In parullel we recommend that OCE investigate alternative mechanisms for in- 
sfitutionalizing career education and make explicit the position that institutionalization of career 
education should not be a major objective of short-term, externally- funded projects, 

GRADE EIGHT RESULTS 

The implementation/process of career education was weakest at the eighth grade level. Eighth 
grade teachers reported less knowledge and understanding and fewer classroom, linkage, and extra 
school activities than either fifth or eleventh grade teachers. We might be inclined to dismiss this 
finding, were it not for the fact that eighth grade students reported less exposure to career education 
activities than did their fifth and eleventh grade counterparts. 

We remain somewhat at a loss to explain these findings. It is possible, certainly, that these results 
are merely an anomaly and would not be replicated in similiar studies; however, the agreement we 
found between teacher implementation and student exposure argues against such a simplistic inter- 
pretation. A second possibility is that our eighth grade sample biased. We checked this possibility 
and determined that the eighth grade samples are not unlike the fifth and eleventh grade samples on 
important demographic variables. A third possibility is that career education is more difficult to im- 
plement at the junior high level. It is this last possibility that deserves serious attention from OCE. If 
taken seriously, this finding has numerous implications for career education projects. We recommend 
that OCE articulate the career exploration stage more clearly, indicating the forms implementation 
could take and explicating expected outcomes. 
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COUNSELOR ROLE 



Counselors displayed the few^t target^nontarget difference amon«; the practitioner groups. 
Examining the frequency with which target counselors engaged in career education activities con- 
firmed our observation that ctunseiors were considerably less active impkmentcan than teachm or 
principals* FHirthermore, adoption strategy was not as important a determinant of implementation 
among counselors as it appeared to be for teachers and principals. 

One explanation for these findings is that some of the ccnmselors' traditional responsibilities were 
usurped by project personnel, leaving the counselor to fill a largely undefined role. It was apparent, 
from our contacts with the projects, that teachers and principals received the bulk of the mservice, but 
it did not appear to us that counselors were as isolated as the questionnaire results suggest. Taken as 
a whole, the counselor results argue persuasively for a more caivful delineation of the counselor's role 
in a career education project 



One constant source of amazement to NTS staff was the yield that project staffs got out of what 
amounted to a rainiscule per student expenditure. The parallel results for teacher implementation 
and student exposure surest that it is pos^ble to realize changes in teacher knowledge, behaviors, 
and attitudes which translate into desired instructional experiences for students. The observed 
process effects are by no means overwhelming; however, it is quite likdy that, given improved process 
measurements, the effects would have been larger. The observe effects are, in fact, likely to be lower- 
bound estimates of the impact of career education on students, teachers, counselors, and principals. It 
is our belief that career education projects similar to those evaluated in this study are accomplishing a 
significant amount of change with a low exi^nditure of resources. It is recommended that OCE study 
the cost-benefit implications of career education, in situations in which benefits are measured initially 
as process effects on practitioners and students. 



We faced a number of difficulties in this study, difficulties stemming from the overall evaluation 
design we used and from the limits of the methodologies available to us. Some of these have been 
discussed extensively in this chapter (i. e., the quality of learner outcome instrumentation and the 
need to further specify career education processes). In this section, difficulties related to sampling 
design, unit of analysis, and level of aggregation are explored. 

Our sampling design made a number of assumptions which constrained the evaluation design. 
These assumptions included presumed demographic similarities among target and nontarget respon- 
dents, low levels of "contamination" of the nontarget sample, and the willingness and ability of 
project staff and practitioners to carry out the test administration program. These assumptions 
proved to be invalid. Nontarget respondents differ markedly from their target peers, particularly on 
the educationally significant ethnicity variable. This required us to conduct analyses by ethnicity as 
well as by treatment group membership, projects, grade, etc . At the same time, differences in treat- 
ment between target and nontarget respondents were not as large as had been expected. Finally, 
biased attrition from pre- to posttest stemming from local decisions concerning test administration 
seriously hampered our ability to compare treatment groups, limited the generalizability of the 
results, and virtually precluded conducting valid overall analyses. 

For these reasons, we reiterate the importance of the recommendation made earlier that, if large- 
scale evaluation is to be successfully implemented, operational responsibilities (ptirticularly in- 
strument administration) must be the exclusive province of the contractor. 

Selecting the unit of analysis for a study such as this dictates numerous subsequent decisions. We 
selected the individual respondent as the most appropriate unit since that unit would provide more 
adequate data for the implementation/process and exposure/outcome analyses. This selection, 
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however, prevented us from conducting rigorous analyses based on other units. Most obviously, we 
could provide only rough estimates of the relationships betwera practitioner reported implementation 
aiUi student reputed exposure. Further, ti^ selected unit allowed few opportimities for ex a minin g the 
rde of the teacher or the school as a catabrst We rgconwiand that studies be fimded which explicitly 
are designed to evaiuate process and outcome interrelationships not based on the individual as tf^ 
unit of analysis. 

We feel that one of the major methodological strengths of this study has been the recognition of 
problems stenmiing from the use of multiple levels of aggregation. In this study we observed 
process/outcome relationships changing, often maztodly, based on the level of aggregation used in the 
analyses. Classroom, building, project, ethnic group, and overall levels were all used where ap- 
propriate during this evaluation, and each level led to somewhat different inferences about career 
education's impacts. Little is known about how changing the level Oi^ aggregation can affect 
evaluation results. It is our impression, however, that insensitivity to the problem has led to many of 
the "no diserved differences due to treatment" reports seen in too many major evaluations. We 
recommend, therefore, that research on level of aggregation be funded in the near future, and, further, 
tlUs research should include reanalyses ofearUer large-scale evaluations. 
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GLOSSARY 



Canonical Redundancy Analysis (CRA) — The extension of multiple regression analyses, when the 
criterion is multivariate. 

CIPP MODEL a model developed by Stufflebeam and others which leads to an examination of the 
context, input, process, and product in an evaluation and which leads to a broadly focused eval- 



Comparison Group Model — presumes that ''trratment*' is discrete, i.e., that career education, per se, 
is a distinct educational methodology. 

Construct Definition - A procedure (or set of (H-ocedures) combining extrinsic definition and intrinsic 
definition to delimit, validate, and label a set of constructs. 

Constructs — hypotheses concerning the nature of organizing influences of observed patterns. 

Determinants — facilitating or impeding forces. 

Determinants of Implementation — a set of thirteen generic and career education-s|^cific contextual 
variables which shape and limit the quality of career education implementation. Examples of 
these determinants include saturation of the adopting organization, time for planning and men- 
tation, organization capacity and history of innovation, adoption strategies, and incentives and 
, disincentives for adoption. 

* 'Early Adoption" — refers to institutions which have readily accepted career education concepts mto 
their programs. 

Exposure to Career Education — The extent to which the individual student participates in career 
education-speciHc activities and is presented with career education*sp0cific concepts. Exposure 
is a function of the quality of implementation and opportunity to learn. 

Extrinsic Definition — procedures, such as determining the face content* for labeling a set of items 
according to their common content. 

Factor Analysis — a process for examining a matrix and expressing the correlations of each variable 
with every other variable. 

Fidelity Perspective — refers to an attempt to assess the discrepancy between a project and non- 
project achievement level in implementation. 

Implementation/Process Model — A conceptual framework which posits a set of determinants of 
implementation which effects the quality of implementation (a function of five process dimen- 
sions). The quality of implementation interacts with opportunity to learn to produce student 
exposxu-e to career education. Exposure, coupled with pretest, determines posttest scores. 

Intrinsic Definition — procedures, such as factor analysis, used to determine the common content 
of responses to sets of items. Used in this study to delimit constructs among the student learner 
outcome and practitioner value internalization items. 

Invariance — structural stability of statistical solutions. 

Joint Effects Model — combines the comparison and process models into one analysis to allow deter- 
mination of the unique effects of treatment group and exposure as well as the common effect 



nation. 
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LEA - local education agency. 

Learner Outcomes - Deaked prtxiuct objectives for students. Three sets of learner outojmw are 
^^taSered in this study: {I) the nine OE leamer outcomes current at «i^?f«?f/.»^«^d£ 
ShTust of nine ]mrJ outcomes devel*^ by NTS adapted from the OCE list ^^^^ ^ 
more readUy operationalized; and (3) a set of thirteen leamer outcome constructs developed by 
NTS through construct definiUon procedures from the pretest results of the outcome instru- 
mentation. 

"Mapping" - The process of breaking down evaluation questions into data elements and data 
sources. 

"Matched" Student Samples - Samples of groups of students for whom parallel test data were 
available from both pre- and posttesting and from process testing. 

MRC — Multiple regression/correlation analysis. 

"Mutual Adaptation" Perepective - an assessment viewpoint that recognizes the evolving character 
of career education and the multipHcity of evaluative concepts that may be used to judge its 
achievements. 

NM5001 to NM5008> - The New Mexico Career Education Test Senes. 
NTS — NTS Research Corporation. 
OCE - Office of Career Education. 

Ohio 6, 8. and 10 - The Ohio Career Development Test Series. 
OMB — Office of Management and Budget. 

Opportunity to Learn - A complex of factors including motivation, attendance, and attitude toward 
school which, in combination with the quaUty of implementation, determines student exposure 
to career education. This component of the Implementation/Process Model was not addressed 
directly in this evaluation. 

PECE — The Program of Exploration in Career Education Knowledge Test 

Power Analysis - seeks to control the error rate of faiHng to detect a treatment effect by determining 
sample sizes needed to reach statistical significance. 

Process Dimensions - A set of five resultants of the determinants of implementation which descril^ 
the quaHty of implementation. The five dimensions include: (1) changes m tangible resources. (2) 
changes in intangible resources. (3) changes in implementation knowledge and understanding. 
(4) changes in implementor role/behavior, and (5) changes in implementor value mtemahzaUon. 

Process Model - presumes that "treatment" is a continuum and that students vary in exposure to 
career education concepts and activities. J 

PSI - "Process Specification Index" - the ratio of the unique effect on the criterion of treatment to 
the uni^ effect of treatment plus the common effect due to process and treatment group. 

Quality of Implementation - the level of implementation of an innovation, described by the five pro- 
cess dimensions. 



i 



RegreMion/Commonality Analysis — procedure whkh partitions the variance in a critarion that is pre- 
dictable from two mme sets of predicts variables into tl^ |p»t>portion that can be uniquely 
associated with ^uih predictw set, and the prc^x»tion that is in common with two or more othsr 
predictor seta. 



SEA — State Education Agex^y. 
SOS — Th€ Self-Observation Scales. 

Type I Error — concluding thaj^ ttore is an effect when there is n<me. 
lype 11 Error — concluding that there is no ^ect when there is one. 
•'Uniquenesses" — first order commonality coefficients. 
USOE - United States Office of Education. 

Value Internalization — implementor commitment and attitudes toward an innovation. 
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